diff --git a/agent/config.yml b/agent/config.yml index 17a72cb..625448b 100644 --- a/agent/config.yml +++ b/agent/config.yml @@ -62,9 +62,9 @@ compaction: - handoff - snapcompact idleThresholdTokens: 100000 - thresholdPercent: 85 + thresholdPercent: 70 idleEnabled: true - thresholdTokens: 150000 + thresholdTokens: -1 interruptMode: immediate defaultThinkingLevel: medium hideThinkingBlock: false diff --git a/agent/managed-skills/compose-to-quadlet-cutover/SKILL.md b/agent/managed-skills/compose-to-quadlet-cutover/SKILL.md new file mode 100644 index 0000000..e4247bc --- /dev/null +++ b/agent/managed-skills/compose-to-quadlet-cutover/SKILL.md @@ -0,0 +1,72 @@ +--- +name: compose-to-quadlet-cutover +description: "Convert a compose-only service to a boot-persistent rootless Quadlet unit on the Dankclaw/Dankserver fleet — checklist plus the traps (CMD-SHELL healthchecks in distroless images, --format docker, EnvironmentFile HOME=/root, loopback publish posture)" +--- + +# Compose service → boot-persistent Quadlet unit (this fleet) + +Use when a service in `dankclaw/compose.yaml` (or `dankserver/`) must survive a +reboot. Precedents to copy: `dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/` +and `dankclaw/observability/quadlet/` (five units, 2026-09-11). + +## Checklist + +1. **Unit file** at `dankclaw//quadlet/.container`, mirroring the + compose service line for line: `Image=`, `ContainerName=` + (Caddy and scrape configs address containers by name), `Network=dankclaw_default`, + `User=0:0` when the image's own user cannot write a rootless-owned bind mount, + `Volume=` for each mount (`:ro` preserved), `Exec=` for the compose `command:`, + `PublishPort=`, `HealthCmd`/`HealthInterval`/`HealthTimeout`/`HealthRetries`/ + `HealthStartPeriod`, `[Service] Restart=always` + `RestartSec=10`, + `[Install] WantedBy=default.target`. Never `systemctl --user enable` the + generated unit — `[Install]` plus linger IS the boot persistence. +2. **Exposure**: publish on `127.0.0.1` unless something off-host genuinely + needs it. Caddy reaches containers by name over the shared network, and the + homepage status timer probes loopback, so the 0.0.0.0 publish is almost + always gratuitous and is a guard-chain bypass. +3. **Handler** in `scripts/deploy-remote.sh`: `install_quadlet ` → + `systemctl --user restart .service` → `wait_for_health` on the + loopback endpoint. Rebuild locally built images first + (`build_compose_image`), and seed any per-service env file (see traps). +4. **Planner**: map `dankclaw//*` and `shared//*` to the service in + `scripts/deploy-plan.py`; keep dependencies ordered in `SERVICE_ORDER`. + Add a `scripts/test-changed.sh` case *above* the generic + `dankclaw/*/quadlet/*.container` fallback. +5. **Retire the compose entry** in the same change — two owners for one + container name is how a stale duplicate comes back. +6. **Seam test** `tests/_quadlet.bats`: boot persistence, restart policy, + no wildcard publish, container name still matching Caddy/scrape references, + health command, deployer installs+restarts+waits, never `rm`, never + `enable`. Migrate any older assertion that read the compose file. +7. **Deploy** per service (`./scripts/deploy.sh dankclaw --service `), + then prove: `is-active`/`is-enabled` = active/generated, container health + `healthy`, the endpoint answers, `list-dependencies default.target` shows the + unit, and off-host probes on the previously published ports now refuse. + +## Traps that cost real time + +- **`HealthCmd` = CMD-SHELL.** podman runs it as `/bin/sh -c`; a distroless + image has no shell, so the check fails forever. `--health-cmd` will not take + exec form either (a JSON array is stored as one literal argument). Put + `HEALTHCHECK ... CMD ["binary","arg"]` in the image and **build with + `--format docker`** — the default OCI format has no HEALTHCHECK field and + drops it with only a warning. +- **Never point a unit's `EnvironmentFile` at the repo root `.env`.** That file + is a gluetun env file carrying `HOME=/root`: systemd sets HOME for the + service, podman then looks for its config under `/root`, and the unit dies + with `stat /root/.config: permission denied`. Use a per-service file under + `/home/dank/Containers//env`, seeded by the deploy handler with a + single-variable filter and validated so a missing secret fails loudly. +- **`Network=host` exporters** that a containerized Prometheus scrapes must + bind the host's LAN address (containers cannot reach the host loopback) plus + `127.0.0.1` for the homepage probe — two `--web.listen-address` flags, never + the wildcard. +- **alloy's UI** listens on the container's loopback and 1.x refuses the `ui` + config block, so the unit must pass `--server.http.listen-addr=0.0.0.0:12345` + before its port can be published; without it the service has no reachable + health endpoint and its status card reads down forever. +- **Locally built images** need `Pull=never` and `Image=localhost//:tag`; + registry-prefixed refs do not resolve from local storage on these hosts. +- Compose-managed containers are replaced in place by `podman run --replace + --rm`; verify afterwards that the container no longer carries + `io.podman.compose.project`. diff --git a/agent/managed-skills/halogen-flash-server-ops/SKILL.md b/agent/managed-skills/halogen-flash-server-ops/SKILL.md new file mode 100644 index 0000000..a164936 --- /dev/null +++ b/agent/managed-skills/halogen-flash-server-ops/SKILL.md @@ -0,0 +1,67 @@ +--- +name: halogen-flash-server-ops +description: "Operate the frank-large halogen-flash-server deployment on Dankclaw (model swaps, restarts, memory sizing, health and client verification, rollback) — use for anything touching frank-large after the 2026-09-11 runtime swap" +--- + +# halogen-flash-server ops (frank-large, Dankclaw) + +frank-large is a **rootless Quadlet container** since 2026-09-11: +`dankclaw/frank-large/quadlet/frank-large.container` in the serverAppConfigs repo, +installed to `~/.config/containers/systemd/`, generated unit `frank-large.service` +in the **user** manager. Full runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`. + +## Never use sudo systemctl on it + +It is a user unit. `sudo systemctl stop frank-large` does nothing useful. + +```bash +systemctl --user status frank-large.service +podman logs frank-large | head -40 # version banner + memory ledger +curl -s http://127.0.0.1:7770/health # 200 only after the engine PONGs +curl -s http://127.0.0.1:7770/v1/models # must list frank-large +``` + +## Deploy / change flags + +1. Edit the `.container` file in the repo (all tunables are inline `Environment=` lines). +2. `./tests/run.sh` (the quadlet seam covers GPU devices, memlock, ipc/seccomp, the single published port, writable weights mount). +3. Commit, push, then `./scripts/deploy.sh dankclaw --service frank-large`. + +The handler retires the old root unit (moving its symlink to the home trash), pulls and +re-tags the image to the `localhost/` alias, installs the unit, restarts it, and gates on +the download when the checkpoint is absent. Never `systemctl --user enable` a generated unit. + +## Sizing a swap + +Read the startup ledger before and after any change: it prints the locked weights, the KV +pool it chose, the working set, the BIOS iGPU carve-out, and the corrected free-memory +figure. `free`, MemAvailable and every monitor built on them overstate free memory by the +size of the model (~68 GiB) because the kernel counts the locked weights as reclaimable +page cache. + +- KV is ~7.2 GiB per 262,144 positions; the prefill arena is 16.7 GiB at `HALOGEN_MAX_TOK=32768`. +- The device budget stops near 47 GiB *with the weights pinned*, and lower on a machine + carrying more besides this server. `HALOGEN_KV_POOL_FIT=1` lowers the pool to fit and + prints what it chose; a refused start needs a smaller `HALOGEN_KV_POOL_POSITIONS` + (`HALOGEN_KV_SLOTS` will not help — slots share one pool). +- Long prompts crawling with the disk busy = file cache, not memory: lower the pool or + raise `HALOGEN_HOST_RESERVE_GIB`. + +## Client contract traps + +- No server-side sampler defaults: a request without `temperature` decodes GREEDY. +- The per-request budget defaults to 8192 and **includes thinking**; a reasoning model + plus a small budget returns EMPTY content with `finish_reason: "length"`. Cap is 65536 + (a 400 above it), and `max_tokens`/`max_completion_tokens`/`max_output_tokens` must agree. +- `response_format` is a 400 ("no constrained decoding") — as it is on FreeToken/frank-small. + Any caller that needs structured output or a tiny budget needs a different backend. +- A request reserves prompt + `max_tokens` positions against the pool; larger requests queue + and then fail with `503 engine_busy` after 3600 s. + +## Verification after any change + +Run a real completion (never just `/health`), a vision probe if the tower is enabled +(image content part with a `data:` URL; `http(s)` URLs are refused), then one call from the +client that consumes it. Rollback: the llama.cpp era is in `.AGENTTRASH/` (`.llama.cpp`), +the STRIX payload and ROCmFPX build stay on disk — but remove the Quadlet from +`~/.config/containers/systemd/` before restoring the root unit, or both claim port 7770. diff --git a/agent/managed-skills/hermes-local-aux-routing/SKILL.md b/agent/managed-skills/hermes-local-aux-routing/SKILL.md new file mode 100644 index 0000000..1bec85a --- /dev/null +++ b/agent/managed-skills/hermes-local-aux-routing/SKILL.md @@ -0,0 +1,87 @@ +--- +name: hermes-local-aux-routing +description: "Fix a broken Hermes auxiliary route (title generation, compression, vision, etc.) when the target is a local OpenAI-compatible backend — diagnose by probing the real call path in-container, then repoint the transport" +--- + +# Hermes auxiliary routing against local backends + +Use when a Hermes feature that calls an auxiliary task fails against a local +endpoint on this fleet (Frank-small on Dankserver :7770, frank-large on Dankclaw +:7770). The usual cause is a request shape the local runtime does not implement, +not a routing mistake. + +## Diagnose with the real call path + +Do not guess from the config. Run the real function inside the container — it +uses the caller's actual kwargs, the config-driven provider resolution and the +real response parsing: + +```bash +base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \ + 'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \ + podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \ + podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py' +``` + +Copy the file AFTER any restart (`--replace --rm` discards the container layer), +and import the app's own constants so the probe cannot drift from production: + +```python +import sys; sys.path.insert(0, "/opt/hermes") +from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text +from agent.auxiliary_client import call_llm +resp = call_llm(task="title_generation", messages=[...], max_tokens=64, + temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT}) +``` + +Pass `api_mode=` / extra `extra_body` keys to A/B a fix before touching config. +(The CLI at `/opt/hermes/hermes` uses the system python and dies on +`No module named 'yaml'`; use `/opt/hermes/.venv/bin/python`.) + +## What actually breaks, and the fix + +- **`response_format` (json_object / json_schema) is a hard 400** on every local + backend here — neither FreeToken nor halogen implements constrained decoding. + Config cannot override it: `call_llm` merges the task's `extra_body` first and + then `update()`s the caller's, so the hardcoded one wins. Only temperature and + max_tokens have reactive retry paths. +- **Fix: move that one task to the Responses transport**, which never forwards + `response_format`: + ```yaml + auxiliary: + title_generation: + provider: dankserver-small + model: frank-small + api_mode: codex_responses # real per-task key + extra_body: + reasoning: + effort: none # mandatory, see below + ``` +- **Disable thinking for any cap-limited task.** The adapter omits reasoning + only when the config says `reasoning.enabled: false`; note that the + `reasoning_effort: none` shorthand produces exactly that, which leaves the + server default (thinking ON) and the 64-token title budget is spent on the + think — empty content, `finish_reason: length`. The explicit + `extra_body.reasoning.effort: none` instead sends + `{"effort":"none","summary":"auto"}`, which the local /v1/responses honors. +- Both local runtimes also answer `/v1/responses` and `/v1/messages`, so no + proxy is needed to use the alternate transport. + +## Verify and land it + +1. Re-run the probe with **no** `api_mode`/reasoning kwargs — it must pass from + config alone; that is the production path. +2. Keep the live file (`/home/dank/Containers/hermes/data/config.yaml`, owned by + uid 109999, needs sudo) and the repo example + `dankclaw/hermes/config.yaml.example` in step; the example is a sanitized + backup and lags otherwise. +3. `./tests/run.sh`, commit, push — a Hermes config change is docs+example only, + no service deploy needed. + +## Gotchas + +- `auxiliary..enabled: false` is checked BEFORE `apply_instant_title`, so + disabling a broken title task also removes the derived title — it does not + degrade gracefully the way a failure does. +- A local call can also fail at the model level: a 200 with an error-text body, + or a chatty non-title answer. Always read the raw content, not the status. diff --git a/agent/managed-skills/noctalia-config-remote-edit/SKILL.md b/agent/managed-skills/noctalia-config-remote-edit/SKILL.md new file mode 100644 index 0000000..980fcc5 --- /dev/null +++ b/agent/managed-skills/noctalia-config-remote-edit/SKILL.md @@ -0,0 +1,28 @@ +--- +name: noctalia-config-remote-edit +description: "Edit live Noctalia v5 shell config on a fleet host over SSH: which config layer is live, why noctalia msg fails from ssh context, and the systemd-run --pipe workaround that works." +--- + +# Noctalia config edit over SSH (v5 native, both fleet hosts) + +## Current architecture (since 2026-09-10 unification) +Both dankware and blankpad run the bare `noctalia` v5 binary as the desktop shell (launch: niri `spawn-at-startup "noctalia"`). The old Quickshell-hosted v4 architecture (`qs -c noctalia-shell`, JSON config) is retired on both hosts. Live config layer = TOML only: +- Hand-edit declarative layer: `~/.config/noctalia/.toml` (e.g. `config.toml`). +- Runtime state file `~/.local/state/noctalia/settings.toml` loads LAST and overrides declarative values for the same keys — never hand-edit it; UI/IPC writes land there. + +## Why msg fails remotely, and the workaround that works +`noctalia msg ` resolves the running instance via the D-Bus session bus. sshd-spawned non-login shells lack `DBUS_SESSION_BUS_ADDRESS`, so plain ssh exec prints `error: noctalia is not running` even when the shell IS running. Do not conclude "shell dead" from this error. +Working remote procedure (runs in systemd user-manager context where the bus address exists): +``` +systemd-run --user --pipe --wait noctalia msg [args] +``` +Verified: `msg status` returns live JSON state; unknown panel ids return an error listing all valid panel ids (clipboard, control-center, launcher, polkit, session, setup-wizard, test, tray-drawer, wallpaper). + +## Verified v5 msg verb grammar (keybinds-safe) +`panel-toggle `, `session lock|suspend|lock-and-suspend|logout|reboot|shutdown`, `volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`, `media next|previous|playPause` (graceful no-op with rc 0 when no active player), `brightness-up [target] [step]`, `brightness-down`, `settings-open [context]`, `settings-toggle [context]`, `config-reload`, `wallpaper-get [connector]`, `wallpaper-set `, `wallpaper-next`. + +## Wallpaper directory semantics +Empty/unset `[wallpaper] directory` resolves to XDG Pictures (`$HOME/Pictures`) via user-dirs.dirs; an explicit directory key overrides. Fleet default pattern: point `directory` at the repo-tracked wallpapers dir so git-tracked wallpapers serve the shell without copying files. After writing config.toml over ssh, the file watcher hot-reloads automatically; `config-reload` via systemd-run --pipe forces it immediately. Note: the *current* wallpaper stays the built-in default asset until the next rotation or an explicit `wallpaper-next`; `wallpaper-get` then reports a path inside the configured directory = confirmation the config took effect. + +## Lock chain (unified) +swayidle-lock.service ExecStart on both hosts: `/usr/bin/swayidle -w timeout 300 '/usr/bin/noctalia msg session lock' before-sleep '...' after-resume '...'`. `msg` resolves via session bus, so no WAYLAND_DISPLAY pinning is needed in unit context (niri's own /proc/PID/environ lacks WAYLAND_DISPLAY but spawn-at-startup children still receive a working display connection empirically — verified by qs-hosted autostart working from boot pre-unification). diff --git a/agent/memories/--home-dank-.omp--/learned.md b/agent/memories/--home-dank-.omp--/learned.md index 55469ab..c128afd 100644 --- a/agent/memories/--home-dank-.omp--/learned.md +++ b/agent/memories/--home-dank-.omp--/learned.md @@ -1,3 +1,7 @@ +- On this host's niri build, niri msg do reload-config is rejected ("unrecognized subcommand 'do'") — verified 2026-09-10 during the noctalia unification deploy. The working config-reload path is kill -USR1 $(pidof niri) (SIGUSR1 to the niri process); verified on both fleet hosts (dankware pid via /proc, blankpad over ssh). Future keybind/config deploys that need live reload must use the SIGUSR1 path, not niri msg do. +- Noctalia fleet architecture UNIFIED 2026-09-10: both dankware and blankpad now run bare noctalia (v5 native, package noctalia 5.0.1-1.1) as the desktop shell, launched via niri spawn-at-startup "noctalia" from repo autostart.kdl. The v4 Quickshell-hosted architecture (qs -c noctalia-shell, JSON settings.json) is retired on dankware; qs host process killed and replaced live by a transient systemd-run unit (noctalia-live) that hands off to autostart at next login — single-instance guard in the binary makes overlap safe. Live config layer on both hosts = TOML only: hand-edit ~/.config/noctalia/config.toml; runtime state ~/.local/state/noctalia/settings.toml loads last and overrides same keys (never hand-edit). Verified mechanics: noctalia msg resolves the running instance via D-Bus session bus, so it works from systemd user-unit context (systemd-run --user --pipe --wait) even though niri's own environ lacks WAYLAND_DISPLAY; spawn-at-startup children do receive a working display connection. Lock keybinds/swayidle unified to noctalia msg session lock; verified verb grammar includes panel-toggle {launcher,session,control-center,...}, volume-up/down/mute, mic-mute, media next/previous/playPause (graceful no-op rc 0 with no active player), brightness-up/down. Wallpapers: wallpaper-next/wallpaper-get confirm rotation draws from the configured [wallpaper] directory on both hosts; current wallpaper stays on the built-in default asset until first rotation after config-reload. Leftover dead packages on dankware (noctalia-shell 4.7.7, noctalia-qs fork) are inert — removal is optional cleanup, zero dependents post-flip. +- Divergence explained (as of 2026-09-10): dankware and blankpad both show noctalia 5.0.1-1.1 but run different live architectures because each host's autostart froze at its adoption-era architecture and package upgrades never rewrote autostart commands. Fossil record from pacman install dates: noctalia-shell 4.7.7-3 installed on dankware Jul 3 (v4 Quickshell-hosted architecture, JSON config; module files owned at /etc/xdg/quickshell/noctalia-shell/, launched via qs -c noctalia-shell from base niri package autostart.kdl); noctalia-qs 0.0.12-2 (Quickshell fork hosting the shell) installed Aug 24; v5-era noctalia package landed Sep 6 as a parallel package owning ONLY /usr/bin/noctalia + desktop file + README — CLI/client surface, no shell module. blankpad was provisioned fresh Sep 9 entirely from CachyOS repo packaging (noctalia 5.0.1-1.1 + vanilla quickshell 0.3.1-1.1 + cachyos-niri-noctalia 1.4.0-1; no v4-era packages exist in that repo set), so its zz-blankpad override autostart launches bare noctalia (native v5, TOML config). Consequence: settings.json is genuinely the live config on dankware (v4 code), while blankpad runs true v5 with TOML state at ~/.config/noctalia/config.toml + ~/.local/state/noctalia/settings.toml. Unification candidate when approved: flip both to noctalia as single entrypoint, but verify first that the v5 packaging on dankware ships the shell module (current -Ql evidence says CLI-only) — a unified flip likely requires installing the full v5 package set on dankware first. Diagnostic recipe for this class of question: query pacman install dates + package ownership (pacman -Qi/-Ql/-Qo, pgrep -af) to reconstruct which architecture each host actually runs rather than trusting version labels. +- Noctalia shell runtime split on the fleet (as of 2026-09): both dankware and blankpad install noctalia 5.0.1-1.1 (-dirty = git build), but launch differently — dankware hosts it via qs -c noctalia-shell (Quickshell host; reads JSON-era config ~/.config/noctalia/settings.json), blankpad runs the bare v5 binary (reads hand-edit ~/.config/noctalia/config.toml, runtime state written to ~/.local/state/noctalia/settings.toml which overrides declarative values last). Lock chain on both hosts: primary qs -c noctalia-shell ipc call lockScreen lock, fallback noctalia msg session lock|suspend|lock-and-suspend|logout|reboot|shutdown. Quirk: noctalia msg ... from a non-session context (ssh command exec) always answers "noctalia is not running" even when the shell runs — detection needs session-scoped state absent in sshd-spawned shells; DBUS_SESSION_BUS_ADDRESS override does NOT fix it. Correct remote procedure: write config files directly (they hot-reload or apply at next start); never treat the msg failure over ssh as evidence the shell is dead. Wallpaper dir default when unconfigured = XDG Pictures ($HOME/Pictures) via user-dirs.dirs; explicit [wallpaper] directory key overrides it — blankpad was seeded with directory = /home/blank/.local/share/noctalia/wallpapers (repo-tracked symlink target). - blank@blankpad (CachyOS, Arch-like) deployment COMPLETED 2026-09-09 (supersedes earlier "pending action awaiting user go-ahead" note): full-replica rsync of /home/dank/.omp/ → /home/blank/.omp/ (.git included; push to Gitea NOT required for this transport) followed by deploy.sh sync on blankpad. SSH was initially blocked there by firewall default-drop (timeout, not refuse) and opened only after the user started sshd locally — expect that blocker pattern again for new fleet hosts. Host facts verified at deploy time: login shell fish; niri runs as niri --session reading the repo-symlinked config.kdl; installed binaries include noctalia CLI (its keybinds override package uses noctalia msg ..., NOT canonical qs -c noctalia-shell ipc call), alacritty, firefox, nautilus, fish, zsh, shelly-notifications; MISSING: qs/quickshell, ghostty, gnome-calculator — canonical keybinds relying on those are silent no-ops if deployed there. ~/.p10k.zsh on blankpad is an inert repo symlink until powerlevel10k-zsh is installed and sourced. zz-blankpad override package (display/keybinds/misc/rules/autostart/.zshrc) is tracked in the repo; deploy.sh's target derivation must strip the nested base-package segment for zz-* packages or overrides land on phantom paths like $HOME/niri/.config/... - blankpad is a fleet machine reached only as blank@blankpad.local (mDNS FQDN; bare hostname does not resolve from Dankware; 192.168.0.83). As of 2026-09-09 it runs the command-center deployment at /home/blank/.omp with host-specific divergence preserved as tracked override package dotfiles/zz-blankpad/: its own display.kdl output layout (not Dankware's DP/eDP rig), autostart/keybinds targeting the bare noctalia CLI instead of qs -c noctalia-shell ipc call, misc.kdl without renderD129, rules.kdl without the Noctalia floating-window rule, and blank's own minimal .zshrc. Its login shell is fish. Initial blocker (port 22 drop-not-refuse = sshd down or firewall default-drop) was fixed same day by user locally; if SSH drops again on blankpad, suspect firewall default-drop first, not auth. - blankpad (user blank) is fleet member at 192.168.0.83, reachable ONLY via mDNS FQDN blankpad.local — bare hostname does not resolve from Dankware. As of 2026-09-09 its SSH was unreachable from the LAN segment: ping fine (~1 ms, wired) but TCP 22 and alt ports (2222/8022/22222/10022) all TIMEOUT = firewall default-drop or sshd down; drop-not-refuse distinguishes this from sshd-listening-but-auth-failing. Pending action awaiting user go-ahead: deploy command-center to blank@blankpad via full-replica rsync of /home/dank/.omp/ (incl .git so core.hooksPath stays armed pre-push) → deploy.sh --check review → sync; transport deliberately chosen over clone-from-Gitea because main was 6 commits ahead of origin/main and push is reserved as a user-only trigger. Note: origin remote on Dankware embeds the Gitea token in plaintext URL — keep it off other machines' .git/config by using rsync as transport. _(context: User asked to set up the dotfiles/command-center deployment on blank@blankpad with a password credential; SSH entry point unreachable from this segment, so work was staged and handed back with exact unblock steps (systemctl enable --now sshd + firewalld internal-zone ssh service).)_ diff --git a/agent/memories/--home-dank-.omp--/raw_memories.md b/agent/memories/--home-dank-.omp--/raw_memories.md index 737764c..5cdefbd 100644 --- a/agent/memories/--home-dank-.omp--/raw_memories.md +++ b/agent/memories/--home-dank-.omp--/raw_memories.md @@ -1,14 +1,25 @@ # Raw Memories ## 01a088e9-5dc1-767c-9a3d-cc0a8a7ec00f -updated_at: 1789026078 -- **Repository (/home/dank/.omp)**: `main` branch is in sync with `origin/main` (0/0 divergence). `develop` branch has an active stash. Uncommitted work includes: `AGENTS.md` (modified, -83 lines), `agent/config.yml` (modified), `agent/models.yml` (modified), and deleted skills under `agent/memories/**/skills/*`. -- **Package Installations (blankpad)**: Installed `quickshell`, `ghostty`, `gnome-calculator`. Dependencies resolved: `gtk4-layer-shell`, `ghostty-shell-integration`, `ghostty-terminfo`, `qt6-wayland`, `cpptrace`, `gtksourceview5`. -- **Nuances**: - - `quickshell` (`/usr/bin/qs`): Keybindings use `qs -c noctalia-shell`, requiring the shell config to be registered under that client ID. - - `ghostty` (`/usr/bin/ghostty`): Zero-config start; auto-sources if `GHOSTTY_RESOURCES_DIR` is set. - - `powerlevel10k-zsh` (Skipped): `~/.zshrc` sources `cachyos-zsh-config`, making the deployed `~/.p10k.zsh` symlink inert decoration until sourced. -- **Shell**: Zsh with `cachyos-zsh-config`. +updated_at: 1789110720 +REPO_STATE: +- Path: /home/dank/.omp | Branch: develop (stashes present), main (synced with origin/main). +- Uncommitted/Tracked: Modified AGENTS.md, agent/config.yml, agent/models.yml, memory files/blobs. +- Deleted: Skills under agent/memories/**/skills/. +- Untracked: Top-level directories and managed skills. + +DESKTOP_UNIFICATION (noctalia v5 / niri): +- Hosts: dankware & blankpad now run bare /usr/bin/noctalia v5 natively. +- Flip Mechanism: Live runtime flip (killed qs -c host, launched bare via spawn-at-startup). Single-instance guard absorbs overlap at next login. +- Lock Chain: swayidle-lock.service unified. ExecStart: /usr/bin/noctalia msg session lock (3 triggers). msg resolves via D-Bus session bus (no WAYLAND_DISPLAY pinning needed in systemd unit context). +- Keybinds: Rewritten to verified v5 msg verbs (panel-toggle, session lock, volume/mic/media/brightness). Config reload uses kill -USR1 $(pidof niri) (this niri build rejects niri msg do reload-config). +- Wallpaper: Rotation pulls from repo-tracked directory (verified wallpaper-next draws from configured path). +- Deploy: Commit 961de3b local on dankware; rsync transport to blankpad verified. + +RESIDUALS/NOTES: +- Dead packages: noctalia-shell 4.7.7 + noctalia-qs on dankware (zero dependents, optional purge). +- Config layer: v5 uses TOML-only. Legacy v4 colors.json/plugins.json are inert; custom palettes/widgets may visually revert to v5 defaults. +- Transient unit noctalia-live runs until logout; autostart takeover is automatic. ## 01a08864-7ec1-70d0-9225-2f08fcb00986 updated_at: 1789003385 @@ -18,7 +29,7 @@ Modes: --check (dry report), --adopt (identicals), sync (default: adopt + link n Hooks: Arms core.hooksPath=.githooks on deploy; self-gating enabled. Secrets: GITEA_TOKEN moved from .zshrc to untracked ~/.config/environment.d/secrets.env (0600). .zshrc has guarded source stub. Untracked: agent/managed-skills/* (auto-learn), secrets.env (local-only). -Target: blank@blankpad (pw: ). Setup: git clone + deploy.sh. +Target: blank@blankpad (pw: Assface.3241). Setup: git clone + deploy.sh. Pending: Push main to origin/main. ## 01a05602-bf3e-76c3-97a1-38e51673bb86 diff --git a/agent/memories/--home-dank-.omp--/rollout_summaries/01a088e9-5dc1-767c-9a3d-cc0a8a7ec00f-omp_repo_status_noct.md b/agent/memories/--home-dank-.omp--/rollout_summaries/01a088e9-5dc1-767c-9a3d-cc0a8a7ec00f-omp_repo_status_noct.md new file mode 100644 index 0000000..3df4eae --- /dev/null +++ b/agent/memories/--home-dank-.omp--/rollout_summaries/01a088e9-5dc1-767c-9a3d-cc0a8a7ec00f-omp_repo_status_noct.md @@ -0,0 +1,4 @@ +thread_id: 01a088e9-5dc1-767c-9a3d-cc0a8a7ec00f +updated_at: 1789110720 + +Repository /home/dank/.omp: develop has stashes, main synced with origin. Uncommitted: modified AGENTS.md, config/models, deleted skills. Desktop unification complete: dankware & blankpad run bare noctalia v5 native via live runtime flip. Lock uses D-Bus msg without WAYLAND_DISPLAY pinning. Keybinds use verified v5 verbs. Config reload via kill -USR1. v5 uses TOML-only (legacy v4 JSONs inert). Commit 961de3b local on dankware; rsync deploy to blankpad verified. diff --git a/agent/memories/--home-dank-.omp--/rollout_summaries/01a088e9-5dc1-767c-9a3d-cc0a8a7ec00f-repo_review_packages.md b/agent/memories/--home-dank-.omp--/rollout_summaries/01a088e9-5dc1-767c-9a3d-cc0a8a7ec00f-repo_review_packages.md deleted file mode 100644 index fd00798..0000000 --- a/agent/memories/--home-dank-.omp--/rollout_summaries/01a088e9-5dc1-767c-9a3d-cc0a8a7ec00f-repo_review_packages.md +++ /dev/null @@ -1,4 +0,0 @@ -thread_id: 01a088e9-5dc1-767c-9a3d-cc0a8a7ec00f -updated_at: 1789026078 - -Reviewed /home/dank/.omp repo: main synced, changes in AGENTS.md/config/models, deleted skills, stash on develop. Installed quickshell, ghostty, gnome-calculator on blankpad; skipped powerlevel10k-zsh (inert in current zshrc). diff --git a/agent/memories/--home-dank-Projects-serverAppConfigs--/learned.md b/agent/memories/--home-dank-Projects-serverAppConfigs--/learned.md index ce5c0c8..46b25b5 100644 --- a/agent/memories/--home-dank-Projects-serverAppConfigs--/learned.md +++ b/agent/memories/--home-dank-Projects-serverAppConfigs--/learned.md @@ -1,3 +1,8 @@ +- Making a Hermes auxiliary call work against a LOCAL OpenAI-compatible backend on this fleet (solved for title generation on 2026-09-11). Symptom: every session logs "Title generation failed" and falls back to a message-derived title. Root cause chain, in the order worth checking: (1) Hermes hardcodes response_format: json_schema on the title call (agent/title_generator.py _TITLE_RESPONSE_FORMAT), and NO local backend implements constrained decoding — frank-small (FreeToken) and frank-large (halogen) both answer 400 response_format json_object/json_schema is not supported (no constrained decoding); (2) config CANNOT override it, because call_llm merges effective_extra_body = _get_task_extra_body(task); effective_extra_body.update(extra_body or {}) — the caller's dict wins; (3) only temperature and max_tokens have reactive retry paths (_is_unsupported_parameter_error is generic but is only wired to those two); (4) auxiliary.task.enabled: false does NOT avoid the failure cleanly — it sits before apply_instant_title, so it removes titles entirely. Working configuration, verified through the real code path: auxiliary.title_generation with api_mode: codex_responses (a real per-task config key, _resolve_ta[REDACTED] reads it) plus extra_body.reasoning.effort: none. The Responses transport builds resp_kwargs and never forwards response_format (it also omits max_tokens/temperature); the reasoning half is mandatory, not decoration — with thinking on, the 64-token cap is spent on the think and content comes back empty with finish_reason=length. reasoning.effort: none maps to {"effort":"none","summary":"auto"} + include, which FreeToken's /v1/responses honors (4 output tokens, 1.5 s). Note reasoning_effort: none as a *shorthand* yields {"enabled": False}, which the adapter treats as "omit reasoning" — leaving the server default (thinking ON) and reintroducing the empty-content failure; use the explicit extra_body.reasoning.effort form. Both local runtimes also expose /v1/respon _(context: Hermes auto-titling on Dankclaw after frank-npu was retired (2026-09-11): the NPU had been the only non-reasoning endpoint, so the title route fell to frank-small and failed on structured output. Verified with in-container probes calling agent.auxiliary_client.call_llm with title_generator's own constants; final config proven end to end (raw {"title": "Friendly greeting"} - 'Friendly greeting'), c)_ +- Compose→Quadlet cutover traps on this fleet (learned converting the five Dankclaw observability containers on 2026-09-11). (1) A unit's HealthCmd reaches podman as **CMD-SHELL** (/bin/sh -c). In a distroless image that silently produces a permanently-unhealthy container: the loki image has no shell, so the busybox command that worked by hand failed as a healthcheck, and podman does NOT accept exec form through --health-cmd (a JSON array is stored as one literal argument). The command has to live in the IMAGE as HEALTHCHECK ... CMD ["binary","arg"], which also means the build must pass --format docker — the default OCI image format has no HEALTHCHECK field and discards it with a warning. (2) Pointing a unit's EnvironmentFile at a repo .env is unsafe: this repo's ROOT .env is a gluetun env file that carries HOME=/root, so systemd sets HOME for the service, podman looks for its config under /root, and the unit dies with stat /root/.config: permission denied. The fleet convention is a per-service file (/home/dank/Containers/svc/env) seeded by the deploy handler with a single-variable awk filter, validated, so a missing secret fails the deploy loudly instead of starting a container whose login cannot complete. (3) For exposure, binding published ports to 127.0.0.1 is sufficient and safe: Caddy reaches containers by name over dankclaw_default and the host-side homepage status timer (/srv/homepage/services.json probes → homepage-status.py) probes loopback, so nothing legitimate needed the 0.0.0.0 publish. The exception is a Network=host exporter that a containerized Prometheus must scrape — it needs the LAN IP (containers cannot reach the host's loopback), so it binds loopback + LAN explicitly and is the only deliberate LAN listener. (4) Convert with matching ContainerName= so Caddy/scrape configs that address containers by name keep working, keep Pull=never for locally built images (loki/alloy), and delete the compose definitions in the same change so only one owner exist _(context: Converting prometheus/grafana/loki/alloy/node-exporter from dankclaw/compose.yaml services to rootless Quadlet units (repo commits dab81b9, fe4eeaa, b96359e, 25c9398, 701ab14), including the fixes found by deploying: the grafana HOME=/root failure, the loki CMD-SHELL/distroless healthcheck failure, and the OCI-vs-docker image format that dropped the HEALTHCHECK.)_ +- Retiring a repo-managed root systemd unit on Dankclaw/Dankserver (the fleet installs them as symlinks: relink_unit does ln -sfn /home/dank/serverAppConfigs/scripts/svc.service /etc/systemd/system/svc.service): systemctl mask svc FAILS with "File '/etc/systemd/system/svc.service' already exists and is a symlink to …" — mask refuses to replace an existing unit file. Correct order, and it also avoids the documented disable-deletes-the-unit-file quirk: (1) systemctl stop, (2) systemctl disable (may or may not delete the file on these hosts — tolerate both), (3) if the file/symlink still exists, mv it into ~/.AGENTTRASH/dated-topic/host-units/ (never rm), (4) mv the /etc/systemd/system/multi-user.target.wants/svc.service enable symlink too or it dangles, (5) systemctl daemon-reload, (6) now systemctl mask svc succeeds and prints "Unit … does not exist, proceeding anyway" then creates the /dev/null symlink. Also: moving the repo file to .AGENTTRASH before the host symlink is cleaned leaves a dangling unit, so do the host side in the same change as the repo deletion. _(context: Retiring the frank-npu systemd unit on Dankclaw as part of the 2026-09-11 frank-large swap; deploy-remote.sh's relink_unit installs root units as symlinks into the repo checkout.)_ +- peonist-ai/halogen-flash-server on Dankclaw (frank-large, deployed 2026-09-11) — traps worth keeping: (1) The podman 5.x quadlet **v1 generator** accepts every key this image needs (AddDevice, PublishPort, PodmanArgs=--ipc=host --security-opt seccomp=unconfined, Ulimit=memlock=-1:-1, HealthCmd, Pull=never) — verified with a throwaway unit before shipping; the generated ExecStart also carries --replace --rm, so every restart discards the container layer (fine when state is on a bind mount). (2) HALOGEN_DOWNLOAD fires only when the checkpoint is absent and requires a WRITABLE /models; the whole 127 GB repo fetched in 542 s (~240 MB/s) anonymously, no token needed. Start-to-health was ~12 s (engine PONG) when the weights were still in page cache, ~50 s after a cold reboot. (3) The startup ledger is the only truthful memory source: 68.0 GiB weights locked + 7.2 GiB KV pool (262144 positions) + 21.1 GiB working set, and free/MemAvailable overstate available memory by the model size because locked pages count as reclaimable cache — Grafana dashboards will be wrong by ~68 GiB. It also detected the host's 2.0 GiB BIOS iGPU carve-out. (4) Serving contract gotchas: no server-side sampler defaults (omitted temperature = GREEDY), the per-request budget defaults to 8192 and INCLUDES thinking (so a reasoning model + small budget returns empty content with finish_reason=length), the cap is 65536, and response_format is a hard 400 ("no constrained decoding"). (5) That last point matters fleet-wide: FreeToken (frank-small) refuses response_format identically, so Hermes's auto-title call (max_tokens=64 + response_format json_schema) fails against BOTH living local backends and falls back to its message-derived title. Any aux route that needs constrained decoding or a tiny budget needs an endpoint with thinking off and grammar support; the retired FLM NPU was that endpoint. _(context: Full frank-large runtime+model swap on Dankclaw 2026-09-11: llama.cpp/kingjones STRIX - halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next w4b, as a rootless Quadlet container on port 7770, plus the amd_iommu=off kernel change that retired the frank-npu endpoint. Repo commit 301914b; runbooks serverAppConfigs-docs/LLM/halogen-flash-server.md and dankclaw-iommu-translation-off.md.)_ +- halogen-flash-server (github.com/peonist-ai/halogen-flash-server) is an OCI-image-only inference server for gfx1151 Strix Halo, evaluated 2026-09-11 as a replacement for the llama.cpp frank-large unit on Dankclaw. Verified facts: - Image ghcr.io/peonist-ai/halogen-flash-server:0.5.8 (latest CHANGELOG tag). Ports: OpenAI-compatible **8731** (publish this), engine **8730** (NO auth — keep unpublished/loopback only). - deploy/entrypoint.sh modes: all (default; both roles in one container), engine, api (split so front-end restarts don't reload the model), bench [drafters] [max_tokens] [effort] [reps]. Engine binary is flash_serve, CLI --ck FILE (not --checkpoint), --slots/--ctx/--max-tok. - Weights: HF peonist-ai/halogen-qwen3.8-flash-next (public, ungated) — qwen38-flash-next-w4b.hgn 115.55 GiB + ck.overlay.hgn 2.31 GiB quality sidecar (auto-picked up; engine warns if absent) + tokenizer/ (8 files). Optional vision.hgn 0.84 GiB, overlay-speed.hgn (trades o_proj calibration). Container-side download via HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next. - Run requirements: --device /dev/kfd --device /dev/dri --group-add keep-groups --security-opt seccomp=unconfined --ipc=host, memlock unlimited, native Linux only (WSL2//dev/dxg unsupported), gfx1151 only (hard-rejected else). Reference SKU = Ryzen AI Max+ 395 / 128 GB — exactly Dankclaw, so the baked HALOGEN_MATMUL_TUNING_FILE does *not* warn. - Key flags (env-only, no config file, read once at startup): HALOGEN_CHECKPOINT (default /models/qwen38-flash-next-w4b.hgn), HALOGEN_CK_OVERLAY, HALOGEN_MODEL_ID (default halogen-qwen3.8-flash-next), HALOGEN_DRAFTER_DEFAULT=1 (MTP speculation on by default, byte-identical at temp 0), HALOGEN_PROMPT_CACHE=2 (numeric; 1 = byte-identical resume), HALOGEN_CACHE_ENTRIES=8, HALOGEN_MAX_TOKENS_CAP=65536, HALOGEN_CTX=262144, HALOGEN_KV_POOL_POSITIONS (default 524288 = 35.0 GB; 262144 = 27.8 GB), HALOGEN_KV_SLOTS=4, HALOGEN_KV_POOL_FIT=1, HALOGEN_HOST_RESERVE_GIB=20, HALOGEN_MAX_TOK= _(context: Research phase of a grilling session: replacing both the model and runtime of frank-large (Dankclaw) with halogen-flash-server, requested by Frank 2026-09-11. All facts read from the upstream repo (README, docs/FLAGS.md, deploy/entrypoint.sh, docker-compose.yml, CHANGELOG), the HF weights repo API, the ghcr image manifest, and live probes of Dankclaw.)_ - On Dankclaw's Hermes stack, hermes-webui ran with the image-baked HERMES_HOME=/opt/data while the gateway used /data; /opt/data is an auto-created anonymous volume at the image's VOLUME point, so every container recreation (restart/redeploy) reset the webui's config home to the image-default template (provider: auto → openrouter.ai with no API key) — this is why model renames "didn't propagate" and chat failed with connect errors. Fix (2026-09-05, commit 184df9a): Environment=HERMES_HOME=/data in the hermes-webui quadlet so webui + gateway share one config/state root. Diagnostic order for "can't access models" on hermes: (1) check which config home each runtime actually reads (podman exec c printenv HERMES_HOME), (2) resolve provider→endpoint with the app's own code (hermes_cli.providers.resolve_user_provider(name, cfg["providers"]) accepts api/url/base_url keys on providers: dict entries), (3) probe the served endpoint with the exact model id. Model ids must match served aliases exactly (frank-large @192.168.0.143:7770, frank-small @192.168.0.235:7770, gemma4-it:e4b @192.168.0.143:7771 — FLM routes only exact registry tags). - Hermes (Dankclaw) model routing as of 2026-09-05: main default = frank-large (Dankclaw 192.168.0.143:7770/v1), small/aux routes = frank-small (Dankserver 192.168.0.235:7770/v1), vision + title_generation route to the exact FLM registry tag gemma4-it:e4b (192.168.0.143:7771/v1). The pre-rename aliases frank-default-large/-small/-sightsound are dead since the 2026-09-03 rename cutover — never route on them. Live config lives at /home/dank/Containers/hermes/data/config.yaml (owned by uid 109999; needs sudo to read/edit); the repo's dankclaw/hermes/config.yaml.example is a sanitized backup and can lag live, so diff live first before editing either. Signal adapter is currently disabled in live config (platforms.signal.enabled: false). - On the guarded franksdank.cloud vhosts (Caddy → Anubis :8923 → Authentik outpost), every unauthenticated request — roots and deep paths alike — is answered with HTTP **200** and an HTML challenge page (titleMaking sure you're not a bot!/title), not a 4xx. Verifying guard health by status code alone is meaningless; assert on leakage markers instead (copyparty UI title, homepage id="grid", services.json "category" JSON shape). Non-JS clients (curl/wget/WebDAV) can never solve the PoW challenge, so they are permanently parked at the challenge page — this is the entire security boundary for copyparty, which serves authenticated traffic with anonymous read perms (-v /w::r:ra,dank). Pinned by tests/e2e/specs/15-unauth-fetch.spec.ts (guest project, empty storage state): any guard-route regression that lets app bytes answer fails loudly; any gate implementation passes. diff --git a/agent/memories/--home-dank-Projects-serverAppConfigs--/raw_memories.md b/agent/memories/--home-dank-Projects-serverAppConfigs--/raw_memories.md index 8ea7d0b..803104e 100644 --- a/agent/memories/--home-dank-Projects-serverAppConfigs--/raw_memories.md +++ b/agent/memories/--home-dank-Projects-serverAppConfigs--/raw_memories.md @@ -1,5 +1,13 @@ # Raw Memories +## 01a07351-8e05-75bc-9deb-bd9447e1256d +updated_at: 1788678884 +ROOT CAUSE: Alias drift (pre-rename IDs persisted) + Config-home drift (HERMES_HOME pointed to /opt/data, an auto-created anonymous volume). Container recreation wipes /opt/data, reverting to image-default template config. +DECISION: Pinned Environment=HERMES_HOME=/data in quadlet unit to unify config and state root under /data. In-place renamed live config IDs to frank-large/frank-small/dankclaw-npu. +WORKFLOWS AND PITFALLS: Podman VOLUME paths get fresh anonymous volumes on recreation; edits lost unless pinned. provider: auto without explicit api: routes to OpenRouter; fails if API key missing. FLM/NPU routes by exact registry tag; HTTP 200 does not guarantee valid model (watch for error: Failed to load tag model!). Named providers in providers: dict resolve via api:/url:/base_url keys; custom: requires custom_providers[] entry. +VERIFICATION: Use resolve_model_provider() and resolve_user_provider() for routing truth; probe endpoints with exact served IDs; /api/providers lists models with key_source: config_yaml. +ARTIFACTS: /data/config.yaml, api/config.py, providers.py, runtime_provider.py, quadlet units, deploy.sh. + ## 01a06ef4-c78b-71df-98e0-e1b930f260a5 updated_at: 1788579402 Category shift: Sonarr moved to 'admin' in shared/homepage/services.json. diff --git a/agent/memories/--home-dank-Projects-serverAppConfigs--/rollout_summaries/01a07351-8e05-75bc-9deb-bd9447e1256d-hermes_model_config_.md b/agent/memories/--home-dank-Projects-serverAppConfigs--/rollout_summaries/01a07351-8e05-75bc-9deb-bd9447e1256d-hermes_model_config_.md new file mode 100644 index 0000000..58ba2cd --- /dev/null +++ b/agent/memories/--home-dank-Projects-serverAppConfigs--/rollout_summaries/01a07351-8e05-75bc-9deb-bd9447e1256d-hermes_model_config_.md @@ -0,0 +1,4 @@ +thread_id: 01a07351-8e05-75bc-9deb-bd9447e1256d +updated_at: 1788678884 + +Resolved Hermes model inaccessibility for frank-small, frank-large, and frank-npu caused by alias drift and HERMES_HOME pointing to an ephemeral anonymous volume. Pinned Environment=HERMES_HOME=/data in quadlet unit to unify config and state root. Resolution relies on explicit api: endpoints in /data/config.yaml. Key pitfalls: container recreation wipes /opt/data volumes, provider: auto defaults to OpenRouter without API key, FLM/NPU requires exact tag routing. Verified via resolve_model_provider and endpoint probes. diff --git a/agent/models.yml b/agent/models.yml index 8c6bf19..8c840db 100644 --- a/agent/models.yml +++ b/agent/models.yml @@ -49,4 +49,3 @@ providers: # Keep prior thinking blocks in the rendered prompt across turns; # halogen honours it top-level (verified: 275-token prompt delta vs false). preserve_thinking: true -