agent: local role repoint + compaction 70%/-1; sync memories and managed skills
This commit is contained in:
Frank
+2
-2
@@ -62,9 +62,9 @@ compaction:
|
||||
- handoff
|
||||
- snapcompact
|
||||
idleThresholdTokens: 100000
|
||||
thresholdPercent: 85
|
||||
thresholdPercent: 70
|
||||
idleEnabled: true
|
||||
thresholdTokens: 150000
|
||||
thresholdTokens: -1
|
||||
interruptMode: immediate
|
||||
defaultThinkingLevel: medium
|
||||
hideThinkingBlock: false
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
---
|
||||
name: compose-to-quadlet-cutover
|
||||
description: "Convert a compose-only service to a boot-persistent rootless Quadlet unit on the Dankclaw/Dankserver fleet — checklist plus the traps (CMD-SHELL healthchecks in distroless images, --format docker, EnvironmentFile HOME=/root, loopback publish posture)"
|
||||
---
|
||||
|
||||
# Compose service → boot-persistent Quadlet unit (this fleet)
|
||||
|
||||
Use when a service in `dankclaw/compose.yaml` (or `dankserver/`) must survive a
|
||||
reboot. Precedents to copy: `dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`
|
||||
and `dankclaw/observability/quadlet/` (five units, 2026-09-11).
|
||||
|
||||
## Checklist
|
||||
|
||||
1. **Unit file** at `dankclaw/<svc>/quadlet/<name>.container`, mirroring the
|
||||
compose service line for line: `Image=`, `ContainerName=<the compose name>`
|
||||
(Caddy and scrape configs address containers by name), `Network=dankclaw_default`,
|
||||
`User=0:0` when the image's own user cannot write a rootless-owned bind mount,
|
||||
`Volume=` for each mount (`:ro` preserved), `Exec=` for the compose `command:`,
|
||||
`PublishPort=`, `HealthCmd`/`HealthInterval`/`HealthTimeout`/`HealthRetries`/
|
||||
`HealthStartPeriod`, `[Service] Restart=always` + `RestartSec=10`,
|
||||
`[Install] WantedBy=default.target`. Never `systemctl --user enable` the
|
||||
generated unit — `[Install]` plus linger IS the boot persistence.
|
||||
2. **Exposure**: publish on `127.0.0.1` unless something off-host genuinely
|
||||
needs it. Caddy reaches containers by name over the shared network, and the
|
||||
homepage status timer probes loopback, so the 0.0.0.0 publish is almost
|
||||
always gratuitous and is a guard-chain bypass.
|
||||
3. **Handler** in `scripts/deploy-remote.sh`: `install_quadlet <dir> <name>` →
|
||||
`systemctl --user restart <name>.service` → `wait_for_health` on the
|
||||
loopback endpoint. Rebuild locally built images first
|
||||
(`build_compose_image`), and seed any per-service env file (see traps).
|
||||
4. **Planner**: map `dankclaw/<svc>/*` and `shared/<svc>/*` to the service in
|
||||
`scripts/deploy-plan.py`; keep dependencies ordered in `SERVICE_ORDER`.
|
||||
Add a `scripts/test-changed.sh` case *above* the generic
|
||||
`dankclaw/*/quadlet/*.container` fallback.
|
||||
5. **Retire the compose entry** in the same change — two owners for one
|
||||
container name is how a stale duplicate comes back.
|
||||
6. **Seam test** `tests/<svc>_quadlet.bats`: boot persistence, restart policy,
|
||||
no wildcard publish, container name still matching Caddy/scrape references,
|
||||
health command, deployer installs+restarts+waits, never `rm`, never
|
||||
`enable`. Migrate any older assertion that read the compose file.
|
||||
7. **Deploy** per service (`./scripts/deploy.sh dankclaw --service <name>`),
|
||||
then prove: `is-active`/`is-enabled` = active/generated, container health
|
||||
`healthy`, the endpoint answers, `list-dependencies default.target` shows the
|
||||
unit, and off-host probes on the previously published ports now refuse.
|
||||
|
||||
## Traps that cost real time
|
||||
|
||||
- **`HealthCmd` = CMD-SHELL.** podman runs it as `/bin/sh -c`; a distroless
|
||||
image has no shell, so the check fails forever. `--health-cmd` will not take
|
||||
exec form either (a JSON array is stored as one literal argument). Put
|
||||
`HEALTHCHECK ... CMD ["binary","arg"]` in the image and **build with
|
||||
`--format docker`** — the default OCI format has no HEALTHCHECK field and
|
||||
drops it with only a warning.
|
||||
- **Never point a unit's `EnvironmentFile` at the repo root `.env`.** That file
|
||||
is a gluetun env file carrying `HOME=/root`: systemd sets HOME for the
|
||||
service, podman then looks for its config under `/root`, and the unit dies
|
||||
with `stat /root/.config: permission denied`. Use a per-service file under
|
||||
`/home/dank/Containers/<svc>/env`, seeded by the deploy handler with a
|
||||
single-variable filter and validated so a missing secret fails loudly.
|
||||
- **`Network=host` exporters** that a containerized Prometheus scrapes must
|
||||
bind the host's LAN address (containers cannot reach the host loopback) plus
|
||||
`127.0.0.1` for the homepage probe — two `--web.listen-address` flags, never
|
||||
the wildcard.
|
||||
- **alloy's UI** listens on the container's loopback and 1.x refuses the `ui`
|
||||
config block, so the unit must pass `--server.http.listen-addr=0.0.0.0:12345`
|
||||
before its port can be published; without it the service has no reachable
|
||||
health endpoint and its status card reads down forever.
|
||||
- **Locally built images** need `Pull=never` and `Image=localhost/<repo>/<name>:tag`;
|
||||
registry-prefixed refs do not resolve from local storage on these hosts.
|
||||
- Compose-managed containers are replaced in place by `podman run --replace
|
||||
--rm`; verify afterwards that the container no longer carries
|
||||
`io.podman.compose.project`.
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
name: halogen-flash-server-ops
|
||||
description: "Operate the frank-large halogen-flash-server deployment on Dankclaw (model swaps, restarts, memory sizing, health and client verification, rollback) — use for anything touching frank-large after the 2026-09-11 runtime swap"
|
||||
---
|
||||
|
||||
# halogen-flash-server ops (frank-large, Dankclaw)
|
||||
|
||||
frank-large is a **rootless Quadlet container** since 2026-09-11:
|
||||
`dankclaw/frank-large/quadlet/frank-large.container` in the serverAppConfigs repo,
|
||||
installed to `~/.config/containers/systemd/`, generated unit `frank-large.service`
|
||||
in the **user** manager. Full runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`.
|
||||
|
||||
## Never use sudo systemctl on it
|
||||
|
||||
It is a user unit. `sudo systemctl stop frank-large` does nothing useful.
|
||||
|
||||
```bash
|
||||
systemctl --user status frank-large.service
|
||||
podman logs frank-large | head -40 # version banner + memory ledger
|
||||
curl -s http://127.0.0.1:7770/health # 200 only after the engine PONGs
|
||||
curl -s http://127.0.0.1:7770/v1/models # must list frank-large
|
||||
```
|
||||
|
||||
## Deploy / change flags
|
||||
|
||||
1. Edit the `.container` file in the repo (all tunables are inline `Environment=` lines).
|
||||
2. `./tests/run.sh` (the quadlet seam covers GPU devices, memlock, ipc/seccomp, the single published port, writable weights mount).
|
||||
3. Commit, push, then `./scripts/deploy.sh dankclaw --service frank-large`.
|
||||
|
||||
The handler retires the old root unit (moving its symlink to the home trash), pulls and
|
||||
re-tags the image to the `localhost/` alias, installs the unit, restarts it, and gates on
|
||||
the download when the checkpoint is absent. Never `systemctl --user enable` a generated unit.
|
||||
|
||||
## Sizing a swap
|
||||
|
||||
Read the startup ledger before and after any change: it prints the locked weights, the KV
|
||||
pool it chose, the working set, the BIOS iGPU carve-out, and the corrected free-memory
|
||||
figure. `free`, MemAvailable and every monitor built on them overstate free memory by the
|
||||
size of the model (~68 GiB) because the kernel counts the locked weights as reclaimable
|
||||
page cache.
|
||||
|
||||
- KV is ~7.2 GiB per 262,144 positions; the prefill arena is 16.7 GiB at `HALOGEN_MAX_TOK=32768`.
|
||||
- The device budget stops near 47 GiB *with the weights pinned*, and lower on a machine
|
||||
carrying more besides this server. `HALOGEN_KV_POOL_FIT=1` lowers the pool to fit and
|
||||
prints what it chose; a refused start needs a smaller `HALOGEN_KV_POOL_POSITIONS`
|
||||
(`HALOGEN_KV_SLOTS` will not help — slots share one pool).
|
||||
- Long prompts crawling with the disk busy = file cache, not memory: lower the pool or
|
||||
raise `HALOGEN_HOST_RESERVE_GIB`.
|
||||
|
||||
## Client contract traps
|
||||
|
||||
- No server-side sampler defaults: a request without `temperature` decodes GREEDY.
|
||||
- The per-request budget defaults to 8192 and **includes thinking**; a reasoning model
|
||||
plus a small budget returns EMPTY content with `finish_reason: "length"`. Cap is 65536
|
||||
(a 400 above it), and `max_tokens`/`max_completion_tokens`/`max_output_tokens` must agree.
|
||||
- `response_format` is a 400 ("no constrained decoding") — as it is on FreeToken/frank-small.
|
||||
Any caller that needs structured output or a tiny budget needs a different backend.
|
||||
- A request reserves prompt + `max_tokens` positions against the pool; larger requests queue
|
||||
and then fail with `503 engine_busy` after 3600 s.
|
||||
|
||||
## Verification after any change
|
||||
|
||||
Run a real completion (never just `/health`), a vision probe if the tower is enabled
|
||||
(image content part with a `data:` URL; `http(s)` URLs are refused), then one call from the
|
||||
client that consumes it. Rollback: the llama.cpp era is in `.AGENTTRASH/` (`.llama.cpp`),
|
||||
the STRIX payload and ROCmFPX build stay on disk — but remove the Quadlet from
|
||||
`~/.config/containers/systemd/` before restoring the root unit, or both claim port 7770.
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
name: hermes-local-aux-routing
|
||||
description: "Fix a broken Hermes auxiliary route (title generation, compression, vision, etc.) when the target is a local OpenAI-compatible backend — diagnose by probing the real call path in-container, then repoint the transport"
|
||||
---
|
||||
|
||||
# Hermes auxiliary routing against local backends
|
||||
|
||||
Use when a Hermes feature that calls an auxiliary task fails against a local
|
||||
endpoint on this fleet (Frank-small on Dankserver :7770, frank-large on Dankclaw
|
||||
:7770). The usual cause is a request shape the local runtime does not implement,
|
||||
not a routing mistake.
|
||||
|
||||
## Diagnose with the real call path
|
||||
|
||||
Do not guess from the config. Run the real function inside the container — it
|
||||
uses the caller's actual kwargs, the config-driven provider resolution and the
|
||||
real response parsing:
|
||||
|
||||
```bash
|
||||
base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \
|
||||
'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \
|
||||
podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \
|
||||
podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py'
|
||||
```
|
||||
|
||||
Copy the file AFTER any restart (`--replace --rm` discards the container layer),
|
||||
and import the app's own constants so the probe cannot drift from production:
|
||||
|
||||
```python
|
||||
import sys; sys.path.insert(0, "/opt/hermes")
|
||||
from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text
|
||||
from agent.auxiliary_client import call_llm
|
||||
resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
|
||||
temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT})
|
||||
```
|
||||
|
||||
Pass `api_mode=` / extra `extra_body` keys to A/B a fix before touching config.
|
||||
(The CLI at `/opt/hermes/hermes` uses the system python and dies on
|
||||
`No module named 'yaml'`; use `/opt/hermes/.venv/bin/python`.)
|
||||
|
||||
## What actually breaks, and the fix
|
||||
|
||||
- **`response_format` (json_object / json_schema) is a hard 400** on every local
|
||||
backend here — neither FreeToken nor halogen implements constrained decoding.
|
||||
Config cannot override it: `call_llm` merges the task's `extra_body` first and
|
||||
then `update()`s the caller's, so the hardcoded one wins. Only temperature and
|
||||
max_tokens have reactive retry paths.
|
||||
- **Fix: move that one task to the Responses transport**, which never forwards
|
||||
`response_format`:
|
||||
```yaml
|
||||
auxiliary:
|
||||
title_generation:
|
||||
provider: dankserver-small
|
||||
model: frank-small
|
||||
api_mode: codex_responses # real per-task key
|
||||
extra_body:
|
||||
reasoning:
|
||||
effort: none # mandatory, see below
|
||||
```
|
||||
- **Disable thinking for any cap-limited task.** The adapter omits reasoning
|
||||
only when the config says `reasoning.enabled: false`; note that the
|
||||
`reasoning_effort: none` shorthand produces exactly that, which leaves the
|
||||
server default (thinking ON) and the 64-token title budget is spent on the
|
||||
think — empty content, `finish_reason: length`. The explicit
|
||||
`extra_body.reasoning.effort: none` instead sends
|
||||
`{"effort":"none","summary":"auto"}`, which the local /v1/responses honors.
|
||||
- Both local runtimes also answer `/v1/responses` and `/v1/messages`, so no
|
||||
proxy is needed to use the alternate transport.
|
||||
|
||||
## Verify and land it
|
||||
|
||||
1. Re-run the probe with **no** `api_mode`/reasoning kwargs — it must pass from
|
||||
config alone; that is the production path.
|
||||
2. Keep the live file (`/home/dank/Containers/hermes/data/config.yaml`, owned by
|
||||
uid 109999, needs sudo) and the repo example
|
||||
`dankclaw/hermes/config.yaml.example` in step; the example is a sanitized
|
||||
backup and lags otherwise.
|
||||
3. `./tests/run.sh`, commit, push — a Hermes config change is docs+example only,
|
||||
no service deploy needed.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- `auxiliary.<task>.enabled: false` is checked BEFORE `apply_instant_title`, so
|
||||
disabling a broken title task also removes the derived title — it does not
|
||||
degrade gracefully the way a failure does.
|
||||
- A local call can also fail at the model level: a 200 with an error-text body,
|
||||
or a chatty non-title answer. Always read the raw content, not the status.
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
name: noctalia-config-remote-edit
|
||||
description: "Edit live Noctalia v5 shell config on a fleet host over SSH: which config layer is live, why noctalia msg fails from ssh context, and the systemd-run --pipe workaround that works."
|
||||
---
|
||||
|
||||
# Noctalia config edit over SSH (v5 native, both fleet hosts)
|
||||
|
||||
## Current architecture (since 2026-09-10 unification)
|
||||
Both dankware and blankpad run the bare `noctalia` v5 binary as the desktop shell (launch: niri `spawn-at-startup "noctalia"`). The old Quickshell-hosted v4 architecture (`qs -c noctalia-shell`, JSON config) is retired on both hosts. Live config layer = TOML only:
|
||||
- Hand-edit declarative layer: `~/.config/noctalia/<name>.toml` (e.g. `config.toml`).
|
||||
- Runtime state file `~/.local/state/noctalia/settings.toml` loads LAST and overrides declarative values for the same keys — never hand-edit it; UI/IPC writes land there.
|
||||
|
||||
## Why msg fails remotely, and the workaround that works
|
||||
`noctalia msg <command>` resolves the running instance via the D-Bus session bus. sshd-spawned non-login shells lack `DBUS_SESSION_BUS_ADDRESS`, so plain ssh exec prints `error: noctalia is not running` even when the shell IS running. Do not conclude "shell dead" from this error.
|
||||
Working remote procedure (runs in systemd user-manager context where the bus address exists):
|
||||
```
|
||||
systemd-run --user --pipe --wait noctalia msg <command> [args]
|
||||
```
|
||||
Verified: `msg status` returns live JSON state; unknown panel ids return an error listing all valid panel ids (clipboard, control-center, launcher, polkit, session, setup-wizard, test, tray-drawer, wallpaper).
|
||||
|
||||
## Verified v5 msg verb grammar (keybinds-safe)
|
||||
`panel-toggle <id>`, `session lock|suspend|lock-and-suspend|logout|reboot|shutdown`, `volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`, `media next|previous|playPause` (graceful no-op with rc 0 when no active player), `brightness-up [target] [step]`, `brightness-down`, `settings-open [context]`, `settings-toggle [context]`, `config-reload`, `wallpaper-get [connector]`, `wallpaper-set <path>`, `wallpaper-next`.
|
||||
|
||||
## Wallpaper directory semantics
|
||||
Empty/unset `[wallpaper] directory` resolves to XDG Pictures (`$HOME/Pictures`) via user-dirs.dirs; an explicit directory key overrides. Fleet default pattern: point `directory` at the repo-tracked wallpapers dir so git-tracked wallpapers serve the shell without copying files. After writing config.toml over ssh, the file watcher hot-reloads automatically; `config-reload` via systemd-run --pipe forces it immediately. Note: the *current* wallpaper stays the built-in default asset until the next rotation or an explicit `wallpaper-next`; `wallpaper-get` then reports a path inside the configured directory = confirmation the config took effect.
|
||||
|
||||
## Lock chain (unified)
|
||||
swayidle-lock.service ExecStart on both hosts: `/usr/bin/swayidle -w timeout 300 '/usr/bin/noctalia msg session lock' before-sleep '...' after-resume '...'`. `msg` resolves via session bus, so no WAYLAND_DISPLAY pinning is needed in unit context (niri's own /proc/PID/environ lacks WAYLAND_DISPLAY but spawn-at-startup children still receive a working display connection empirically — verified by qs-hosted autostart working from boot pre-unification).
|
||||
@@ -1,3 +1,7 @@
|
||||
- On this host's niri build, niri msg do reload-config is rejected ("unrecognized subcommand 'do'") — verified 2026-09-10 during the noctalia unification deploy. The working config-reload path is kill -USR1 $(pidof niri) (SIGUSR1 to the niri process); verified on both fleet hosts (dankware pid via /proc, blankpad over ssh). Future keybind/config deploys that need live reload must use the SIGUSR1 path, not niri msg do.
|
||||
- Noctalia fleet architecture UNIFIED 2026-09-10: both dankware and blankpad now run bare noctalia (v5 native, package noctalia 5.0.1-1.1) as the desktop shell, launched via niri spawn-at-startup "noctalia" from repo autostart.kdl. The v4 Quickshell-hosted architecture (qs -c noctalia-shell, JSON settings.json) is retired on dankware; qs host process killed and replaced live by a transient systemd-run unit (noctalia-live) that hands off to autostart at next login — single-instance guard in the binary makes overlap safe. Live config layer on both hosts = TOML only: hand-edit ~/.config/noctalia/config.toml; runtime state ~/.local/state/noctalia/settings.toml loads last and overrides same keys (never hand-edit). Verified mechanics: noctalia msg resolves the running instance via D-Bus session bus, so it works from systemd user-unit context (systemd-run --user --pipe --wait) even though niri's own environ lacks WAYLAND_DISPLAY; spawn-at-startup children do receive a working display connection. Lock keybinds/swayidle unified to noctalia msg session lock; verified verb grammar includes panel-toggle {launcher,session,control-center,...}, volume-up/down/mute, mic-mute, media next/previous/playPause (graceful no-op rc 0 with no active player), brightness-up/down. Wallpapers: wallpaper-next/wallpaper-get confirm rotation draws from the configured [wallpaper] directory on both hosts; current wallpaper stays on the built-in default asset until first rotation after config-reload. Leftover dead packages on dankware (noctalia-shell 4.7.7, noctalia-qs fork) are inert — removal is optional cleanup, zero dependents post-flip.
|
||||
- Divergence explained (as of 2026-09-10): dankware and blankpad both show noctalia 5.0.1-1.1 but run different live architectures because each host's autostart froze at its adoption-era architecture and package upgrades never rewrote autostart commands. Fossil record from pacman install dates: noctalia-shell 4.7.7-3 installed on dankware Jul 3 (v4 Quickshell-hosted architecture, JSON config; module files owned at /etc/xdg/quickshell/noctalia-shell/, launched via qs -c noctalia-shell from base niri package autostart.kdl); noctalia-qs 0.0.12-2 (Quickshell fork hosting the shell) installed Aug 24; v5-era noctalia package landed Sep 6 as a parallel package owning ONLY /usr/bin/noctalia + desktop file + README — CLI/client surface, no shell module. blankpad was provisioned fresh Sep 9 entirely from CachyOS repo packaging (noctalia 5.0.1-1.1 + vanilla quickshell 0.3.1-1.1 + cachyos-niri-noctalia 1.4.0-1; no v4-era packages exist in that repo set), so its zz-blankpad override autostart launches bare noctalia (native v5, TOML config). Consequence: settings.json is genuinely the live config on dankware (v4 code), while blankpad runs true v5 with TOML state at ~/.config/noctalia/config.toml + ~/.local/state/noctalia/settings.toml. Unification candidate when approved: flip both to noctalia as single entrypoint, but verify first that the v5 packaging on dankware ships the shell module (current -Ql evidence says CLI-only) — a unified flip likely requires installing the full v5 package set on dankware first. Diagnostic recipe for this class of question: query pacman install dates + package ownership (pacman -Qi/-Ql/-Qo, pgrep -af) to reconstruct which architecture each host actually runs rather than trusting version labels.
|
||||
- Noctalia shell runtime split on the fleet (as of 2026-09): both dankware and blankpad install noctalia 5.0.1-1.1 (-dirty = git build), but launch differently — dankware hosts it via qs -c noctalia-shell (Quickshell host; reads JSON-era config ~/.config/noctalia/settings.json), blankpad runs the bare v5 binary (reads hand-edit ~/.config/noctalia/config.toml, runtime state written to ~/.local/state/noctalia/settings.toml which overrides declarative values last). Lock chain on both hosts: primary qs -c noctalia-shell ipc call lockScreen lock, fallback noctalia msg session lock|suspend|lock-and-suspend|logout|reboot|shutdown. Quirk: noctalia msg ... from a non-session context (ssh command exec) always answers "noctalia is not running" even when the shell runs — detection needs session-scoped state absent in sshd-spawned shells; DBUS_SESSION_BUS_ADDRESS override does NOT fix it. Correct remote procedure: write config files directly (they hot-reload or apply at next start); never treat the msg failure over ssh as evidence the shell is dead. Wallpaper dir default when unconfigured = XDG Pictures ($HOME/Pictures) via user-dirs.dirs; explicit [wallpaper] directory key overrides it — blankpad was seeded with directory = /home/blank/.local/share/noctalia/wallpapers (repo-tracked symlink target).
|
||||
- blank@blankpad (CachyOS, Arch-like) deployment COMPLETED 2026-09-09 (supersedes earlier "pending action awaiting user go-ahead" note): full-replica rsync of /home/dank/.omp/ → /home/blank/.omp/ (.git included; push to Gitea NOT required for this transport) followed by deploy.sh sync on blankpad. SSH was initially blocked there by firewall default-drop (timeout, not refuse) and opened only after the user started sshd locally — expect that blocker pattern again for new fleet hosts. Host facts verified at deploy time: login shell fish; niri runs as niri --session reading the repo-symlinked config.kdl; installed binaries include noctalia CLI (its keybinds override package uses noctalia msg ..., NOT canonical qs -c noctalia-shell ipc call), alacritty, firefox, nautilus, fish, zsh, shelly-notifications; MISSING: qs/quickshell, ghostty, gnome-calculator — canonical keybinds relying on those are silent no-ops if deployed there. ~/.p10k.zsh on blankpad is an inert repo symlink until powerlevel10k-zsh is installed and sourced. zz-blankpad override package (display/keybinds/misc/rules/autostart/.zshrc) is tracked in the repo; deploy.sh's target derivation must strip the nested base-package segment for zz-* packages or overrides land on phantom paths like $HOME/niri/.config/...
|
||||
- blankpad is a fleet machine reached only as blank@blankpad.local (mDNS FQDN; bare hostname does not resolve from Dankware; 192.168.0.83). As of 2026-09-09 it runs the command-center deployment at /home/blank/.omp with host-specific divergence preserved as tracked override package dotfiles/zz-blankpad/: its own display.kdl output layout (not Dankware's DP/eDP rig), autostart/keybinds targeting the bare noctalia CLI instead of qs -c noctalia-shell ipc call, misc.kdl without renderD129, rules.kdl without the Noctalia floating-window rule, and blank's own minimal .zshrc. Its login shell is fish. Initial blocker (port 22 drop-not-refuse = sshd down or firewall default-drop) was fixed same day by user locally; if SSH drops again on blankpad, suspect firewall default-drop first, not auth.
|
||||
- blankpad (user blank) is fleet member at 192.168.0.83, reachable ONLY via mDNS FQDN blankpad.local — bare hostname does not resolve from Dankware. As of 2026-09-09 its SSH was unreachable from the LAN segment: ping fine (~1 ms, wired) but TCP 22 and alt ports (2222/8022/22222/10022) all TIMEOUT = firewall default-drop or sshd down; drop-not-refuse distinguishes this from sshd-listening-but-auth-failing. Pending action awaiting user go-ahead: deploy command-center to blank@blankpad via full-replica rsync of /home/dank/.omp/ (incl .git so core.hooksPath stays armed pre-push) → deploy.sh --check review → sync; transport deliberately chosen over clone-from-Gitea because main was 6 commits ahead of origin/main and push is reserved as a user-only trigger. Note: origin remote on Dankware embeds the Gitea token in plaintext URL — keep it off other machines' .git/config by using rsync as transport. _(context: User asked to set up the dotfiles/command-center deployment on blank@blankpad with a password credential; SSH entry point unreachable from this segment, so work was staged and handed back with exact unblock steps (systemctl enable --now sshd + firewalld internal-zone ssh service).)_
|
||||
|
||||
@@ -1,14 +1,25 @@
|
||||
# Raw Memories
|
||||
|
||||
## 01a088e9-5dc1-767c-9a3d-cc0a8a7ec00f
|
||||
updated_at: 1789026078
|
||||
- **Repository (/home/dank/.omp)**: `main` branch is in sync with `origin/main` (0/0 divergence). `develop` branch has an active stash. Uncommitted work includes: `AGENTS.md` (modified, -83 lines), `agent/config.yml` (modified), `agent/models.yml` (modified), and deleted skills under `agent/memories/**/skills/*`.
|
||||
- **Package Installations (blankpad)**: Installed `quickshell`, `ghostty`, `gnome-calculator`. Dependencies resolved: `gtk4-layer-shell`, `ghostty-shell-integration`, `ghostty-terminfo`, `qt6-wayland`, `cpptrace`, `gtksourceview5`.
|
||||
- **Nuances**:
|
||||
- `quickshell` (`/usr/bin/qs`): Keybindings use `qs -c noctalia-shell`, requiring the shell config to be registered under that client ID.
|
||||
- `ghostty` (`/usr/bin/ghostty`): Zero-config start; auto-sources if `GHOSTTY_RESOURCES_DIR` is set.
|
||||
- `powerlevel10k-zsh` (Skipped): `~/.zshrc` sources `cachyos-zsh-config`, making the deployed `~/.p10k.zsh` symlink inert decoration until sourced.
|
||||
- **Shell**: Zsh with `cachyos-zsh-config`.
|
||||
updated_at: 1789110720
|
||||
REPO_STATE:
|
||||
- Path: /home/dank/.omp | Branch: develop (stashes present), main (synced with origin/main).
|
||||
- Uncommitted/Tracked: Modified AGENTS.md, agent/config.yml, agent/models.yml, memory files/blobs.
|
||||
- Deleted: Skills under agent/memories/**/skills/.
|
||||
- Untracked: Top-level directories and managed skills.
|
||||
|
||||
DESKTOP_UNIFICATION (noctalia v5 / niri):
|
||||
- Hosts: dankware & blankpad now run bare /usr/bin/noctalia v5 natively.
|
||||
- Flip Mechanism: Live runtime flip (killed qs -c host, launched bare via spawn-at-startup). Single-instance guard absorbs overlap at next login.
|
||||
- Lock Chain: swayidle-lock.service unified. ExecStart: /usr/bin/noctalia msg session lock (3 triggers). msg resolves via D-Bus session bus (no WAYLAND_DISPLAY pinning needed in systemd unit context).
|
||||
- Keybinds: Rewritten to verified v5 msg verbs (panel-toggle, session lock, volume/mic/media/brightness). Config reload uses kill -USR1 $(pidof niri) (this niri build rejects niri msg do reload-config).
|
||||
- Wallpaper: Rotation pulls from repo-tracked directory (verified wallpaper-next draws from configured path).
|
||||
- Deploy: Commit 961de3b local on dankware; rsync transport to blankpad verified.
|
||||
|
||||
RESIDUALS/NOTES:
|
||||
- Dead packages: noctalia-shell 4.7.7 + noctalia-qs on dankware (zero dependents, optional purge).
|
||||
- Config layer: v5 uses TOML-only. Legacy v4 colors.json/plugins.json are inert; custom palettes/widgets may visually revert to v5 defaults.
|
||||
- Transient unit noctalia-live runs until logout; autostart takeover is automatic.
|
||||
|
||||
## 01a08864-7ec1-70d0-9225-2f08fcb00986
|
||||
updated_at: 1789003385
|
||||
@@ -18,7 +29,7 @@ Modes: --check (dry report), --adopt (identicals), sync (default: adopt + link n
|
||||
Hooks: Arms core.hooksPath=.githooks on deploy; self-gating enabled.
|
||||
Secrets: GITEA_TOKEN moved from .zshrc to untracked ~/.config/environment.d/secrets.env (0600). .zshrc has guarded source stub.
|
||||
Untracked: agent/managed-skills/* (auto-learn), secrets.env (local-only).
|
||||
Target: blank@blankpad (pw: <REDACTED-rotated-2026-09-10>). Setup: git clone + deploy.sh.
|
||||
Target: blank@blankpad (pw: Assface.3241). Setup: git clone + deploy.sh.
|
||||
Pending: Push main to origin/main.
|
||||
|
||||
## 01a05602-bf3e-76c3-97a1-38e51673bb86
|
||||
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
thread_id: 01a088e9-5dc1-767c-9a3d-cc0a8a7ec00f
|
||||
updated_at: 1789110720
|
||||
|
||||
Repository /home/dank/.omp: develop has stashes, main synced with origin. Uncommitted: modified AGENTS.md, config/models, deleted skills. Desktop unification complete: dankware & blankpad run bare noctalia v5 native via live runtime flip. Lock uses D-Bus msg without WAYLAND_DISPLAY pinning. Keybinds use verified v5 verbs. Config reload via kill -USR1. v5 uses TOML-only (legacy v4 JSONs inert). Commit 961de3b local on dankware; rsync deploy to blankpad verified.
|
||||
-4
@@ -1,4 +0,0 @@
|
||||
thread_id: 01a088e9-5dc1-767c-9a3d-cc0a8a7ec00f
|
||||
updated_at: 1789026078
|
||||
|
||||
Reviewed /home/dank/.omp repo: main synced, changes in AGENTS.md/config/models, deleted skills, stash on develop. Installed quickshell, ghostty, gnome-calculator on blankpad; skipped powerlevel10k-zsh (inert in current zshrc).
|
||||
@@ -1,3 +1,8 @@
|
||||
- Making a Hermes auxiliary call work against a LOCAL OpenAI-compatible backend on this fleet (solved for title generation on 2026-09-11). Symptom: every session logs "Title generation failed" and falls back to a message-derived title. Root cause chain, in the order worth checking: (1) Hermes hardcodes response_format: json_schema on the title call (agent/title_generator.py _TITLE_RESPONSE_FORMAT), and NO local backend implements constrained decoding — frank-small (FreeToken) and frank-large (halogen) both answer 400 response_format json_object/json_schema is not supported (no constrained decoding); (2) config CANNOT override it, because call_llm merges effective_extra_body = _get_task_extra_body(task); effective_extra_body.update(extra_body or {}) — the caller's dict wins; (3) only temperature and max_tokens have reactive retry paths (_is_unsupported_parameter_error is generic but is only wired to those two); (4) auxiliary.task.enabled: false does NOT avoid the failure cleanly — it sits before apply_instant_title, so it removes titles entirely. Working configuration, verified through the real code path: auxiliary.title_generation with api_mode: codex_responses (a real per-task config key, _resolve_ta[REDACTED] reads it) plus extra_body.reasoning.effort: none. The Responses transport builds resp_kwargs and never forwards response_format (it also omits max_tokens/temperature); the reasoning half is mandatory, not decoration — with thinking on, the 64-token cap is spent on the think and content comes back empty with finish_reason=length. reasoning.effort: none maps to {"effort":"none","summary":"auto"} + include, which FreeToken's /v1/responses honors (4 output tokens, 1.5 s). Note reasoning_effort: none as a *shorthand* yields {"enabled": False}, which the adapter treats as "omit reasoning" — leaving the server default (thinking ON) and reintroducing the empty-content failure; use the explicit extra_body.reasoning.effort form. Both local runtimes also expose /v1/respon _(context: Hermes auto-titling on Dankclaw after frank-npu was retired (2026-09-11): the NPU had been the only non-reasoning endpoint, so the title route fell to frank-small and failed on structured output. Verified with in-container probes calling agent.auxiliary_client.call_llm with title_generator's own constants; final config proven end to end (raw {"title": "Friendly greeting"} - 'Friendly greeting'), c)_
|
||||
- Compose→Quadlet cutover traps on this fleet (learned converting the five Dankclaw observability containers on 2026-09-11). (1) A unit's HealthCmd reaches podman as **CMD-SHELL** (/bin/sh -c). In a distroless image that silently produces a permanently-unhealthy container: the loki image has no shell, so the busybox command that worked by hand failed as a healthcheck, and podman does NOT accept exec form through --health-cmd (a JSON array is stored as one literal argument). The command has to live in the IMAGE as HEALTHCHECK ... CMD ["binary","arg"], which also means the build must pass --format docker — the default OCI image format has no HEALTHCHECK field and discards it with a warning. (2) Pointing a unit's EnvironmentFile at a repo .env is unsafe: this repo's ROOT .env is a gluetun env file that carries HOME=/root, so systemd sets HOME for the service, podman looks for its config under /root, and the unit dies with stat /root/.config: permission denied. The fleet convention is a per-service file (/home/dank/Containers/svc/env) seeded by the deploy handler with a single-variable awk filter, validated, so a missing secret fails the deploy loudly instead of starting a container whose login cannot complete. (3) For exposure, binding published ports to 127.0.0.1 is sufficient and safe: Caddy reaches containers by name over dankclaw_default and the host-side homepage status timer (/srv/homepage/services.json probes → homepage-status.py) probes loopback, so nothing legitimate needed the 0.0.0.0 publish. The exception is a Network=host exporter that a containerized Prometheus must scrape — it needs the LAN IP (containers cannot reach the host's loopback), so it binds loopback + LAN explicitly and is the only deliberate LAN listener. (4) Convert with matching ContainerName= so Caddy/scrape configs that address containers by name keep working, keep Pull=never for locally built images (loki/alloy), and delete the compose definitions in the same change so only one owner exist _(context: Converting prometheus/grafana/loki/alloy/node-exporter from dankclaw/compose.yaml services to rootless Quadlet units (repo commits dab81b9, fe4eeaa, b96359e, 25c9398, 701ab14), including the fixes found by deploying: the grafana HOME=/root failure, the loki CMD-SHELL/distroless healthcheck failure, and the OCI-vs-docker image format that dropped the HEALTHCHECK.)_
|
||||
- Retiring a repo-managed root systemd unit on Dankclaw/Dankserver (the fleet installs them as symlinks: relink_unit does ln -sfn /home/dank/serverAppConfigs/scripts/svc.service /etc/systemd/system/svc.service): systemctl mask svc FAILS with "File '/etc/systemd/system/svc.service' already exists and is a symlink to …" — mask refuses to replace an existing unit file. Correct order, and it also avoids the documented disable-deletes-the-unit-file quirk: (1) systemctl stop, (2) systemctl disable (may or may not delete the file on these hosts — tolerate both), (3) if the file/symlink still exists, mv it into ~/.AGENTTRASH/dated-topic/host-units/ (never rm), (4) mv the /etc/systemd/system/multi-user.target.wants/svc.service enable symlink too or it dangles, (5) systemctl daemon-reload, (6) now systemctl mask svc succeeds and prints "Unit … does not exist, proceeding anyway" then creates the /dev/null symlink. Also: moving the repo file to .AGENTTRASH before the host symlink is cleaned leaves a dangling unit, so do the host side in the same change as the repo deletion. _(context: Retiring the frank-npu systemd unit on Dankclaw as part of the 2026-09-11 frank-large swap; deploy-remote.sh's relink_unit installs root units as symlinks into the repo checkout.)_
|
||||
- peonist-ai/halogen-flash-server on Dankclaw (frank-large, deployed 2026-09-11) — traps worth keeping: (1) The podman 5.x quadlet **v1 generator** accepts every key this image needs (AddDevice, PublishPort, PodmanArgs=--ipc=host --security-opt seccomp=unconfined, Ulimit=memlock=-1:-1, HealthCmd, Pull=never) — verified with a throwaway unit before shipping; the generated ExecStart also carries --replace --rm, so every restart discards the container layer (fine when state is on a bind mount). (2) HALOGEN_DOWNLOAD fires only when the checkpoint is absent and requires a WRITABLE /models; the whole 127 GB repo fetched in 542 s (~240 MB/s) anonymously, no token needed. Start-to-health was ~12 s (engine PONG) when the weights were still in page cache, ~50 s after a cold reboot. (3) The startup ledger is the only truthful memory source: 68.0 GiB weights locked + 7.2 GiB KV pool (262144 positions) + 21.1 GiB working set, and free/MemAvailable overstate available memory by the model size because locked pages count as reclaimable cache — Grafana dashboards will be wrong by ~68 GiB. It also detected the host's 2.0 GiB BIOS iGPU carve-out. (4) Serving contract gotchas: no server-side sampler defaults (omitted temperature = GREEDY), the per-request budget defaults to 8192 and INCLUDES thinking (so a reasoning model + small budget returns empty content with finish_reason=length), the cap is 65536, and response_format is a hard 400 ("no constrained decoding"). (5) That last point matters fleet-wide: FreeToken (frank-small) refuses response_format identically, so Hermes's auto-title call (max_tokens=64 + response_format json_schema) fails against BOTH living local backends and falls back to its message-derived title. Any aux route that needs constrained decoding or a tiny budget needs an endpoint with thinking off and grammar support; the retired FLM NPU was that endpoint. _(context: Full frank-large runtime+model swap on Dankclaw 2026-09-11: llama.cpp/kingjones STRIX - halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next w4b, as a rootless Quadlet container on port 7770, plus the amd_iommu=off kernel change that retired the frank-npu endpoint. Repo commit 301914b; runbooks serverAppConfigs-docs/LLM/halogen-flash-server.md and dankclaw-iommu-translation-off.md.)_
|
||||
- halogen-flash-server (github.com/peonist-ai/halogen-flash-server) is an OCI-image-only inference server for gfx1151 Strix Halo, evaluated 2026-09-11 as a replacement for the llama.cpp frank-large unit on Dankclaw. Verified facts: - Image ghcr.io/peonist-ai/halogen-flash-server:0.5.8 (latest CHANGELOG tag). Ports: OpenAI-compatible **8731** (publish this), engine **8730** (NO auth — keep unpublished/loopback only). - deploy/entrypoint.sh modes: all (default; both roles in one container), engine, api (split so front-end restarts don't reload the model), bench [drafters] [max_tokens] [effort] [reps]. Engine binary is flash_serve, CLI --ck FILE (not --checkpoint), --slots/--ctx/--max-tok. - Weights: HF peonist-ai/halogen-qwen3.8-flash-next (public, ungated) — qwen38-flash-next-w4b.hgn 115.55 GiB + ck.overlay.hgn 2.31 GiB quality sidecar (auto-picked up; engine warns if absent) + tokenizer/ (8 files). Optional vision.hgn 0.84 GiB, overlay-speed.hgn (trades o_proj calibration). Container-side download via HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next. - Run requirements: --device /dev/kfd --device /dev/dri --group-add keep-groups --security-opt seccomp=unconfined --ipc=host, memlock unlimited, native Linux only (WSL2//dev/dxg unsupported), gfx1151 only (hard-rejected else). Reference SKU = Ryzen AI Max+ 395 / 128 GB — exactly Dankclaw, so the baked HALOGEN_MATMUL_TUNING_FILE does *not* warn. - Key flags (env-only, no config file, read once at startup): HALOGEN_CHECKPOINT (default /models/qwen38-flash-next-w4b.hgn), HALOGEN_CK_OVERLAY, HALOGEN_MODEL_ID (default halogen-qwen3.8-flash-next), HALOGEN_DRAFTER_DEFAULT=1 (MTP speculation on by default, byte-identical at temp 0), HALOGEN_PROMPT_CACHE=2 (numeric; 1 = byte-identical resume), HALOGEN_CACHE_ENTRIES=8, HALOGEN_MAX_TOKENS_CAP=65536, HALOGEN_CTX=262144, HALOGEN_KV_POOL_POSITIONS (default 524288 = 35.0 GB; 262144 = 27.8 GB), HALOGEN_KV_SLOTS=4, HALOGEN_KV_POOL_FIT=1, HALOGEN_HOST_RESERVE_GIB=20, HALOGEN_MAX_TOK= _(context: Research phase of a grilling session: replacing both the model and runtime of frank-large (Dankclaw) with halogen-flash-server, requested by Frank 2026-09-11. All facts read from the upstream repo (README, docs/FLAGS.md, deploy/entrypoint.sh, docker-compose.yml, CHANGELOG), the HF weights repo API, the ghcr image manifest, and live probes of Dankclaw.)_
|
||||
- On Dankclaw's Hermes stack, hermes-webui ran with the image-baked HERMES_HOME=/opt/data while the gateway used /data; /opt/data is an auto-created anonymous volume at the image's VOLUME point, so every container recreation (restart/redeploy) reset the webui's config home to the image-default template (provider: auto → openrouter.ai with no API key) — this is why model renames "didn't propagate" and chat failed with connect errors. Fix (2026-09-05, commit 184df9a): Environment=HERMES_HOME=/data in the hermes-webui quadlet so webui + gateway share one config/state root. Diagnostic order for "can't access models" on hermes: (1) check which config home each runtime actually reads (podman exec c printenv HERMES_HOME), (2) resolve provider→endpoint with the app's own code (hermes_cli.providers.resolve_user_provider(name, cfg["providers"]) accepts api/url/base_url keys on providers: dict entries), (3) probe the served endpoint with the exact model id. Model ids must match served aliases exactly (frank-large @192.168.0.143:7770, frank-small @192.168.0.235:7770, gemma4-it:e4b @192.168.0.143:7771 — FLM routes only exact registry tags).
|
||||
- Hermes (Dankclaw) model routing as of 2026-09-05: main default = frank-large (Dankclaw 192.168.0.143:7770/v1), small/aux routes = frank-small (Dankserver 192.168.0.235:7770/v1), vision + title_generation route to the exact FLM registry tag gemma4-it:e4b (192.168.0.143:7771/v1). The pre-rename aliases frank-default-large/-small/-sightsound are dead since the 2026-09-03 rename cutover — never route on them. Live config lives at /home/dank/Containers/hermes/data/config.yaml (owned by uid 109999; needs sudo to read/edit); the repo's dankclaw/hermes/config.yaml.example is a sanitized backup and can lag live, so diff live first before editing either. Signal adapter is currently disabled in live config (platforms.signal.enabled: false).
|
||||
- On the guarded franksdank.cloud vhosts (Caddy → Anubis :8923 → Authentik outpost), every unauthenticated request — roots and deep paths alike — is answered with HTTP **200** and an HTML challenge page (titleMaking sure you're not a bot!/title), not a 4xx. Verifying guard health by status code alone is meaningless; assert on leakage markers instead (copyparty UI title, homepage id="grid", services.json "category" JSON shape). Non-JS clients (curl/wget/WebDAV) can never solve the PoW challenge, so they are permanently parked at the challenge page — this is the entire security boundary for copyparty, which serves authenticated traffic with anonymous read perms (-v /w::r:ra,dank). Pinned by tests/e2e/specs/15-unauth-fetch.spec.ts (guest project, empty storage state): any guard-route regression that lets app bytes answer fails loudly; any gate implementation passes.
|
||||
|
||||
@@ -1,5 +1,13 @@
|
||||
# Raw Memories
|
||||
|
||||
## 01a07351-8e05-75bc-9deb-bd9447e1256d
|
||||
updated_at: 1788678884
|
||||
ROOT CAUSE: Alias drift (pre-rename IDs persisted) + Config-home drift (HERMES_HOME pointed to /opt/data, an auto-created anonymous volume). Container recreation wipes /opt/data, reverting to image-default template config.
|
||||
DECISION: Pinned Environment=HERMES_HOME=/data in quadlet unit to unify config and state root under /data. In-place renamed live config IDs to frank-large/frank-small/dankclaw-npu.
|
||||
WORKFLOWS AND PITFALLS: Podman VOLUME paths get fresh anonymous volumes on recreation; edits lost unless pinned. provider: auto without explicit api: routes to OpenRouter; fails if API key missing. FLM/NPU routes by exact registry tag; HTTP 200 does not guarantee valid model (watch for error: Failed to load tag model!). Named providers in providers: dict resolve via api:/url:/base_url keys; custom:<slug> requires custom_providers[] entry.
|
||||
VERIFICATION: Use resolve_model_provider() and resolve_user_provider() for routing truth; probe endpoints with exact served IDs; /api/providers lists models with key_source: config_yaml.
|
||||
ARTIFACTS: /data/config.yaml, api/config.py, providers.py, runtime_provider.py, quadlet units, deploy.sh.
|
||||
|
||||
## 01a06ef4-c78b-71df-98e0-e1b930f260a5
|
||||
updated_at: 1788579402
|
||||
Category shift: Sonarr moved to 'admin' in shared/homepage/services.json.
|
||||
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
thread_id: 01a07351-8e05-75bc-9deb-bd9447e1256d
|
||||
updated_at: 1788678884
|
||||
|
||||
Resolved Hermes model inaccessibility for frank-small, frank-large, and frank-npu caused by alias drift and HERMES_HOME pointing to an ephemeral anonymous volume. Pinned Environment=HERMES_HOME=/data in quadlet unit to unify config and state root. Resolution relies on explicit api: endpoints in /data/config.yaml. Key pitfalls: container recreation wipes /opt/data volumes, provider: auto defaults to OpenRouter without API key, FLM/NPU requires exact tag routing. Verified via resolve_model_provider and endpoint probes.
|
||||
@@ -49,4 +49,3 @@ providers:
|
||||
# Keep prior thinking blocks in the rendered prompt across turns;
|
||||
# halogen honours it top-level (verified: 275-token prompt delta vs false).
|
||||
preserve_thinking: true
|
||||
|
||||
|
||||
Reference in New Issue
Block a user