agent: local role repoint + compaction 70%/-1; sync memories and managed skills
This commit is contained in:
Frank
@@ -0,0 +1,72 @@
|
||||
---
|
||||
name: compose-to-quadlet-cutover
|
||||
description: "Convert a compose-only service to a boot-persistent rootless Quadlet unit on the Dankclaw/Dankserver fleet — checklist plus the traps (CMD-SHELL healthchecks in distroless images, --format docker, EnvironmentFile HOME=/root, loopback publish posture)"
|
||||
---
|
||||
|
||||
# Compose service → boot-persistent Quadlet unit (this fleet)
|
||||
|
||||
Use when a service in `dankclaw/compose.yaml` (or `dankserver/`) must survive a
|
||||
reboot. Precedents to copy: `dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`
|
||||
and `dankclaw/observability/quadlet/` (five units, 2026-09-11).
|
||||
|
||||
## Checklist
|
||||
|
||||
1. **Unit file** at `dankclaw/<svc>/quadlet/<name>.container`, mirroring the
|
||||
compose service line for line: `Image=`, `ContainerName=<the compose name>`
|
||||
(Caddy and scrape configs address containers by name), `Network=dankclaw_default`,
|
||||
`User=0:0` when the image's own user cannot write a rootless-owned bind mount,
|
||||
`Volume=` for each mount (`:ro` preserved), `Exec=` for the compose `command:`,
|
||||
`PublishPort=`, `HealthCmd`/`HealthInterval`/`HealthTimeout`/`HealthRetries`/
|
||||
`HealthStartPeriod`, `[Service] Restart=always` + `RestartSec=10`,
|
||||
`[Install] WantedBy=default.target`. Never `systemctl --user enable` the
|
||||
generated unit — `[Install]` plus linger IS the boot persistence.
|
||||
2. **Exposure**: publish on `127.0.0.1` unless something off-host genuinely
|
||||
needs it. Caddy reaches containers by name over the shared network, and the
|
||||
homepage status timer probes loopback, so the 0.0.0.0 publish is almost
|
||||
always gratuitous and is a guard-chain bypass.
|
||||
3. **Handler** in `scripts/deploy-remote.sh`: `install_quadlet <dir> <name>` →
|
||||
`systemctl --user restart <name>.service` → `wait_for_health` on the
|
||||
loopback endpoint. Rebuild locally built images first
|
||||
(`build_compose_image`), and seed any per-service env file (see traps).
|
||||
4. **Planner**: map `dankclaw/<svc>/*` and `shared/<svc>/*` to the service in
|
||||
`scripts/deploy-plan.py`; keep dependencies ordered in `SERVICE_ORDER`.
|
||||
Add a `scripts/test-changed.sh` case *above* the generic
|
||||
`dankclaw/*/quadlet/*.container` fallback.
|
||||
5. **Retire the compose entry** in the same change — two owners for one
|
||||
container name is how a stale duplicate comes back.
|
||||
6. **Seam test** `tests/<svc>_quadlet.bats`: boot persistence, restart policy,
|
||||
no wildcard publish, container name still matching Caddy/scrape references,
|
||||
health command, deployer installs+restarts+waits, never `rm`, never
|
||||
`enable`. Migrate any older assertion that read the compose file.
|
||||
7. **Deploy** per service (`./scripts/deploy.sh dankclaw --service <name>`),
|
||||
then prove: `is-active`/`is-enabled` = active/generated, container health
|
||||
`healthy`, the endpoint answers, `list-dependencies default.target` shows the
|
||||
unit, and off-host probes on the previously published ports now refuse.
|
||||
|
||||
## Traps that cost real time
|
||||
|
||||
- **`HealthCmd` = CMD-SHELL.** podman runs it as `/bin/sh -c`; a distroless
|
||||
image has no shell, so the check fails forever. `--health-cmd` will not take
|
||||
exec form either (a JSON array is stored as one literal argument). Put
|
||||
`HEALTHCHECK ... CMD ["binary","arg"]` in the image and **build with
|
||||
`--format docker`** — the default OCI format has no HEALTHCHECK field and
|
||||
drops it with only a warning.
|
||||
- **Never point a unit's `EnvironmentFile` at the repo root `.env`.** That file
|
||||
is a gluetun env file carrying `HOME=/root`: systemd sets HOME for the
|
||||
service, podman then looks for its config under `/root`, and the unit dies
|
||||
with `stat /root/.config: permission denied`. Use a per-service file under
|
||||
`/home/dank/Containers/<svc>/env`, seeded by the deploy handler with a
|
||||
single-variable filter and validated so a missing secret fails loudly.
|
||||
- **`Network=host` exporters** that a containerized Prometheus scrapes must
|
||||
bind the host's LAN address (containers cannot reach the host loopback) plus
|
||||
`127.0.0.1` for the homepage probe — two `--web.listen-address` flags, never
|
||||
the wildcard.
|
||||
- **alloy's UI** listens on the container's loopback and 1.x refuses the `ui`
|
||||
config block, so the unit must pass `--server.http.listen-addr=0.0.0.0:12345`
|
||||
before its port can be published; without it the service has no reachable
|
||||
health endpoint and its status card reads down forever.
|
||||
- **Locally built images** need `Pull=never` and `Image=localhost/<repo>/<name>:tag`;
|
||||
registry-prefixed refs do not resolve from local storage on these hosts.
|
||||
- Compose-managed containers are replaced in place by `podman run --replace
|
||||
--rm`; verify afterwards that the container no longer carries
|
||||
`io.podman.compose.project`.
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
name: halogen-flash-server-ops
|
||||
description: "Operate the frank-large halogen-flash-server deployment on Dankclaw (model swaps, restarts, memory sizing, health and client verification, rollback) — use for anything touching frank-large after the 2026-09-11 runtime swap"
|
||||
---
|
||||
|
||||
# halogen-flash-server ops (frank-large, Dankclaw)
|
||||
|
||||
frank-large is a **rootless Quadlet container** since 2026-09-11:
|
||||
`dankclaw/frank-large/quadlet/frank-large.container` in the serverAppConfigs repo,
|
||||
installed to `~/.config/containers/systemd/`, generated unit `frank-large.service`
|
||||
in the **user** manager. Full runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`.
|
||||
|
||||
## Never use sudo systemctl on it
|
||||
|
||||
It is a user unit. `sudo systemctl stop frank-large` does nothing useful.
|
||||
|
||||
```bash
|
||||
systemctl --user status frank-large.service
|
||||
podman logs frank-large | head -40 # version banner + memory ledger
|
||||
curl -s http://127.0.0.1:7770/health # 200 only after the engine PONGs
|
||||
curl -s http://127.0.0.1:7770/v1/models # must list frank-large
|
||||
```
|
||||
|
||||
## Deploy / change flags
|
||||
|
||||
1. Edit the `.container` file in the repo (all tunables are inline `Environment=` lines).
|
||||
2. `./tests/run.sh` (the quadlet seam covers GPU devices, memlock, ipc/seccomp, the single published port, writable weights mount).
|
||||
3. Commit, push, then `./scripts/deploy.sh dankclaw --service frank-large`.
|
||||
|
||||
The handler retires the old root unit (moving its symlink to the home trash), pulls and
|
||||
re-tags the image to the `localhost/` alias, installs the unit, restarts it, and gates on
|
||||
the download when the checkpoint is absent. Never `systemctl --user enable` a generated unit.
|
||||
|
||||
## Sizing a swap
|
||||
|
||||
Read the startup ledger before and after any change: it prints the locked weights, the KV
|
||||
pool it chose, the working set, the BIOS iGPU carve-out, and the corrected free-memory
|
||||
figure. `free`, MemAvailable and every monitor built on them overstate free memory by the
|
||||
size of the model (~68 GiB) because the kernel counts the locked weights as reclaimable
|
||||
page cache.
|
||||
|
||||
- KV is ~7.2 GiB per 262,144 positions; the prefill arena is 16.7 GiB at `HALOGEN_MAX_TOK=32768`.
|
||||
- The device budget stops near 47 GiB *with the weights pinned*, and lower on a machine
|
||||
carrying more besides this server. `HALOGEN_KV_POOL_FIT=1` lowers the pool to fit and
|
||||
prints what it chose; a refused start needs a smaller `HALOGEN_KV_POOL_POSITIONS`
|
||||
(`HALOGEN_KV_SLOTS` will not help — slots share one pool).
|
||||
- Long prompts crawling with the disk busy = file cache, not memory: lower the pool or
|
||||
raise `HALOGEN_HOST_RESERVE_GIB`.
|
||||
|
||||
## Client contract traps
|
||||
|
||||
- No server-side sampler defaults: a request without `temperature` decodes GREEDY.
|
||||
- The per-request budget defaults to 8192 and **includes thinking**; a reasoning model
|
||||
plus a small budget returns EMPTY content with `finish_reason: "length"`. Cap is 65536
|
||||
(a 400 above it), and `max_tokens`/`max_completion_tokens`/`max_output_tokens` must agree.
|
||||
- `response_format` is a 400 ("no constrained decoding") — as it is on FreeToken/frank-small.
|
||||
Any caller that needs structured output or a tiny budget needs a different backend.
|
||||
- A request reserves prompt + `max_tokens` positions against the pool; larger requests queue
|
||||
and then fail with `503 engine_busy` after 3600 s.
|
||||
|
||||
## Verification after any change
|
||||
|
||||
Run a real completion (never just `/health`), a vision probe if the tower is enabled
|
||||
(image content part with a `data:` URL; `http(s)` URLs are refused), then one call from the
|
||||
client that consumes it. Rollback: the llama.cpp era is in `.AGENTTRASH/` (`.llama.cpp`),
|
||||
the STRIX payload and ROCmFPX build stay on disk — but remove the Quadlet from
|
||||
`~/.config/containers/systemd/` before restoring the root unit, or both claim port 7770.
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
name: hermes-local-aux-routing
|
||||
description: "Fix a broken Hermes auxiliary route (title generation, compression, vision, etc.) when the target is a local OpenAI-compatible backend — diagnose by probing the real call path in-container, then repoint the transport"
|
||||
---
|
||||
|
||||
# Hermes auxiliary routing against local backends
|
||||
|
||||
Use when a Hermes feature that calls an auxiliary task fails against a local
|
||||
endpoint on this fleet (Frank-small on Dankserver :7770, frank-large on Dankclaw
|
||||
:7770). The usual cause is a request shape the local runtime does not implement,
|
||||
not a routing mistake.
|
||||
|
||||
## Diagnose with the real call path
|
||||
|
||||
Do not guess from the config. Run the real function inside the container — it
|
||||
uses the caller's actual kwargs, the config-driven provider resolution and the
|
||||
real response parsing:
|
||||
|
||||
```bash
|
||||
base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \
|
||||
'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \
|
||||
podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \
|
||||
podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py'
|
||||
```
|
||||
|
||||
Copy the file AFTER any restart (`--replace --rm` discards the container layer),
|
||||
and import the app's own constants so the probe cannot drift from production:
|
||||
|
||||
```python
|
||||
import sys; sys.path.insert(0, "/opt/hermes")
|
||||
from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text
|
||||
from agent.auxiliary_client import call_llm
|
||||
resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
|
||||
temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT})
|
||||
```
|
||||
|
||||
Pass `api_mode=` / extra `extra_body` keys to A/B a fix before touching config.
|
||||
(The CLI at `/opt/hermes/hermes` uses the system python and dies on
|
||||
`No module named 'yaml'`; use `/opt/hermes/.venv/bin/python`.)
|
||||
|
||||
## What actually breaks, and the fix
|
||||
|
||||
- **`response_format` (json_object / json_schema) is a hard 400** on every local
|
||||
backend here — neither FreeToken nor halogen implements constrained decoding.
|
||||
Config cannot override it: `call_llm` merges the task's `extra_body` first and
|
||||
then `update()`s the caller's, so the hardcoded one wins. Only temperature and
|
||||
max_tokens have reactive retry paths.
|
||||
- **Fix: move that one task to the Responses transport**, which never forwards
|
||||
`response_format`:
|
||||
```yaml
|
||||
auxiliary:
|
||||
title_generation:
|
||||
provider: dankserver-small
|
||||
model: frank-small
|
||||
api_mode: codex_responses # real per-task key
|
||||
extra_body:
|
||||
reasoning:
|
||||
effort: none # mandatory, see below
|
||||
```
|
||||
- **Disable thinking for any cap-limited task.** The adapter omits reasoning
|
||||
only when the config says `reasoning.enabled: false`; note that the
|
||||
`reasoning_effort: none` shorthand produces exactly that, which leaves the
|
||||
server default (thinking ON) and the 64-token title budget is spent on the
|
||||
think — empty content, `finish_reason: length`. The explicit
|
||||
`extra_body.reasoning.effort: none` instead sends
|
||||
`{"effort":"none","summary":"auto"}`, which the local /v1/responses honors.
|
||||
- Both local runtimes also answer `/v1/responses` and `/v1/messages`, so no
|
||||
proxy is needed to use the alternate transport.
|
||||
|
||||
## Verify and land it
|
||||
|
||||
1. Re-run the probe with **no** `api_mode`/reasoning kwargs — it must pass from
|
||||
config alone; that is the production path.
|
||||
2. Keep the live file (`/home/dank/Containers/hermes/data/config.yaml`, owned by
|
||||
uid 109999, needs sudo) and the repo example
|
||||
`dankclaw/hermes/config.yaml.example` in step; the example is a sanitized
|
||||
backup and lags otherwise.
|
||||
3. `./tests/run.sh`, commit, push — a Hermes config change is docs+example only,
|
||||
no service deploy needed.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- `auxiliary.<task>.enabled: false` is checked BEFORE `apply_instant_title`, so
|
||||
disabling a broken title task also removes the derived title — it does not
|
||||
degrade gracefully the way a failure does.
|
||||
- A local call can also fail at the model level: a 200 with an error-text body,
|
||||
or a chatty non-title answer. Always read the raw content, not the status.
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
name: noctalia-config-remote-edit
|
||||
description: "Edit live Noctalia v5 shell config on a fleet host over SSH: which config layer is live, why noctalia msg fails from ssh context, and the systemd-run --pipe workaround that works."
|
||||
---
|
||||
|
||||
# Noctalia config edit over SSH (v5 native, both fleet hosts)
|
||||
|
||||
## Current architecture (since 2026-09-10 unification)
|
||||
Both dankware and blankpad run the bare `noctalia` v5 binary as the desktop shell (launch: niri `spawn-at-startup "noctalia"`). The old Quickshell-hosted v4 architecture (`qs -c noctalia-shell`, JSON config) is retired on both hosts. Live config layer = TOML only:
|
||||
- Hand-edit declarative layer: `~/.config/noctalia/<name>.toml` (e.g. `config.toml`).
|
||||
- Runtime state file `~/.local/state/noctalia/settings.toml` loads LAST and overrides declarative values for the same keys — never hand-edit it; UI/IPC writes land there.
|
||||
|
||||
## Why msg fails remotely, and the workaround that works
|
||||
`noctalia msg <command>` resolves the running instance via the D-Bus session bus. sshd-spawned non-login shells lack `DBUS_SESSION_BUS_ADDRESS`, so plain ssh exec prints `error: noctalia is not running` even when the shell IS running. Do not conclude "shell dead" from this error.
|
||||
Working remote procedure (runs in systemd user-manager context where the bus address exists):
|
||||
```
|
||||
systemd-run --user --pipe --wait noctalia msg <command> [args]
|
||||
```
|
||||
Verified: `msg status` returns live JSON state; unknown panel ids return an error listing all valid panel ids (clipboard, control-center, launcher, polkit, session, setup-wizard, test, tray-drawer, wallpaper).
|
||||
|
||||
## Verified v5 msg verb grammar (keybinds-safe)
|
||||
`panel-toggle <id>`, `session lock|suspend|lock-and-suspend|logout|reboot|shutdown`, `volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`, `media next|previous|playPause` (graceful no-op with rc 0 when no active player), `brightness-up [target] [step]`, `brightness-down`, `settings-open [context]`, `settings-toggle [context]`, `config-reload`, `wallpaper-get [connector]`, `wallpaper-set <path>`, `wallpaper-next`.
|
||||
|
||||
## Wallpaper directory semantics
|
||||
Empty/unset `[wallpaper] directory` resolves to XDG Pictures (`$HOME/Pictures`) via user-dirs.dirs; an explicit directory key overrides. Fleet default pattern: point `directory` at the repo-tracked wallpapers dir so git-tracked wallpapers serve the shell without copying files. After writing config.toml over ssh, the file watcher hot-reloads automatically; `config-reload` via systemd-run --pipe forces it immediately. Note: the *current* wallpaper stays the built-in default asset until the next rotation or an explicit `wallpaper-next`; `wallpaper-get` then reports a path inside the configured directory = confirmation the config took effect.
|
||||
|
||||
## Lock chain (unified)
|
||||
swayidle-lock.service ExecStart on both hosts: `/usr/bin/swayidle -w timeout 300 '/usr/bin/noctalia msg session lock' before-sleep '...' after-resume '...'`. `msg` resolves via session bus, so no WAYLAND_DISPLAY pinning is needed in unit context (niri's own /proc/PID/environ lacks WAYLAND_DISPLAY but spawn-at-startup children still receive a working display connection empirically — verified by qs-hosted autostart working from boot pre-unification).
|
||||
Reference in New Issue
Block a user