agent: local role repoint + compaction 70%/-1; sync memories and managed skills

agent/memories/--home-dank-Projects-serverAppConfigs--/learned.md

@@ -1,3 +1,8 @@
+- Making a Hermes auxiliary call work against a LOCAL OpenAI-compatible backend on this fleet (solved for title generation on 2026-09-11). Symptom: every session logs "Title generation failed" and falls back to a message-derived title. Root cause chain, in the order worth checking: (1) Hermes hardcodes response_format: json_schema on the title call (agent/title_generator.py _TITLE_RESPONSE_FORMAT), and NO local backend implements constrained decoding — frank-small (FreeToken) and frank-large (halogen) both answer 400 response_format json_object/json_schema is not supported (no constrained decoding); (2) config CANNOT override it, because call_llm merges effective_extra_body = _get_task_extra_body(task); effective_extra_body.update(extra_body or {}) — the caller's dict wins; (3) only temperature and max_tokens have reactive retry paths (_is_unsupported_parameter_error is generic but is only wired to those two); (4) auxiliary.task.enabled: false does NOT avoid the failure cleanly — it sits before apply_instant_title, so it removes titles entirely. Working configuration, verified through the real code path: auxiliary.title_generation with api_mode: codex_responses (a real per-task config key, _resolve_ta[REDACTED] reads it) plus extra_body.reasoning.effort: none. The Responses transport builds resp_kwargs and never forwards response_format (it also omits max_tokens/temperature); the reasoning half is mandatory, not decoration — with thinking on, the 64-token cap is spent on the think and content comes back empty with finish_reason=length. reasoning.effort: none maps to {"effort":"none","summary":"auto"} + include, which FreeToken's /v1/responses honors (4 output tokens, 1.5 s). Note reasoning_effort: none as a *shorthand* yields {"enabled": False}, which the adapter treats as "omit reasoning" — leaving the server default (thinking ON) and reintroducing the empty-content failure; use the explicit extra_body.reasoning.effort form. Both local runtimes also expose /v1/respon _(context: Hermes auto-titling on Dankclaw after frank-npu was retired (2026-09-11): the NPU had been the only non-reasoning endpoint, so the title route fell to frank-small and failed on structured output. Verified with in-container probes calling agent.auxiliary_client.call_llm with title_generator's own constants; final config proven end to end (raw {"title": "Friendly greeting"} - 'Friendly greeting'), c)_
+- Compose→Quadlet cutover traps on this fleet (learned converting the five Dankclaw observability containers on 2026-09-11). (1) A unit's HealthCmd reaches podman as **CMD-SHELL** (/bin/sh -c). In a distroless image that silently produces a permanently-unhealthy container: the loki image has no shell, so the busybox command that worked by hand failed as a healthcheck, and podman does NOT accept exec form through --health-cmd (a JSON array is stored as one literal argument). The command has to live in the IMAGE as HEALTHCHECK ... CMD ["binary","arg"], which also means the build must pass --format docker — the default OCI image format has no HEALTHCHECK field and discards it with a warning. (2) Pointing a unit's EnvironmentFile at a repo .env is unsafe: this repo's ROOT .env is a gluetun env file that carries HOME=/root, so systemd sets HOME for the service, podman looks for its config under /root, and the unit dies with stat /root/.config: permission denied. The fleet convention is a per-service file (/home/dank/Containers/svc/env) seeded by the deploy handler with a single-variable awk filter, validated, so a missing secret fails the deploy loudly instead of starting a container whose login cannot complete. (3) For exposure, binding published ports to 127.0.0.1 is sufficient and safe: Caddy reaches containers by name over dankclaw_default and the host-side homepage status timer (/srv/homepage/services.json probes → homepage-status.py) probes loopback, so nothing legitimate needed the 0.0.0.0 publish. The exception is a Network=host exporter that a containerized Prometheus must scrape — it needs the LAN IP (containers cannot reach the host's loopback), so it binds loopback + LAN explicitly and is the only deliberate LAN listener. (4) Convert with matching ContainerName= so Caddy/scrape configs that address containers by name keep working, keep Pull=never for locally built images (loki/alloy), and delete the compose definitions in the same change so only one owner exist _(context: Converting prometheus/grafana/loki/alloy/node-exporter from dankclaw/compose.yaml services to rootless Quadlet units (repo commits dab81b9, fe4eeaa, b96359e, 25c9398, 701ab14), including the fixes found by deploying: the grafana HOME=/root failure, the loki CMD-SHELL/distroless healthcheck failure, and the OCI-vs-docker image format that dropped the HEALTHCHECK.)_
+- Retiring a repo-managed root systemd unit on Dankclaw/Dankserver (the fleet installs them as symlinks: relink_unit does ln -sfn /home/dank/serverAppConfigs/scripts/svc.service /etc/systemd/system/svc.service): systemctl mask svc FAILS with "File '/etc/systemd/system/svc.service' already exists and is a symlink to …" — mask refuses to replace an existing unit file. Correct order, and it also avoids the documented disable-deletes-the-unit-file quirk: (1) systemctl stop, (2) systemctl disable (may or may not delete the file on these hosts — tolerate both), (3) if the file/symlink still exists, mv it into ~/.AGENTTRASH/dated-topic/host-units/ (never rm), (4) mv the /etc/systemd/system/multi-user.target.wants/svc.service enable symlink too or it dangles, (5) systemctl daemon-reload, (6) now systemctl mask svc succeeds and prints "Unit … does not exist, proceeding anyway" then creates the /dev/null symlink. Also: moving the repo file to .AGENTTRASH before the host symlink is cleaned leaves a dangling unit, so do the host side in the same change as the repo deletion. _(context: Retiring the frank-npu systemd unit on Dankclaw as part of the 2026-09-11 frank-large swap; deploy-remote.sh's relink_unit installs root units as symlinks into the repo checkout.)_
+- peonist-ai/halogen-flash-server on Dankclaw (frank-large, deployed 2026-09-11) — traps worth keeping: (1) The podman 5.x quadlet **v1 generator** accepts every key this image needs (AddDevice, PublishPort, PodmanArgs=--ipc=host --security-opt seccomp=unconfined, Ulimit=memlock=-1:-1, HealthCmd, Pull=never) — verified with a throwaway unit before shipping; the generated ExecStart also carries --replace --rm, so every restart discards the container layer (fine when state is on a bind mount). (2) HALOGEN_DOWNLOAD fires only when the checkpoint is absent and requires a WRITABLE /models; the whole 127 GB repo fetched in 542 s (~240 MB/s) anonymously, no token needed. Start-to-health was ~12 s (engine PONG) when the weights were still in page cache, ~50 s after a cold reboot. (3) The startup ledger is the only truthful memory source: 68.0 GiB weights locked + 7.2 GiB KV pool (262144 positions) + 21.1 GiB working set, and free/MemAvailable overstate available memory by the model size because locked pages count as reclaimable cache — Grafana dashboards will be wrong by ~68 GiB. It also detected the host's 2.0 GiB BIOS iGPU carve-out. (4) Serving contract gotchas: no server-side sampler defaults (omitted temperature = GREEDY), the per-request budget defaults to 8192 and INCLUDES thinking (so a reasoning model + small budget returns empty content with finish_reason=length), the cap is 65536, and response_format is a hard 400 ("no constrained decoding"). (5) That last point matters fleet-wide: FreeToken (frank-small) refuses response_format identically, so Hermes's auto-title call (max_tokens=64 + response_format json_schema) fails against BOTH living local backends and falls back to its message-derived title. Any aux route that needs constrained decoding or a tiny budget needs an endpoint with thinking off and grammar support; the retired FLM NPU was that endpoint. _(context: Full frank-large runtime+model swap on Dankclaw 2026-09-11: llama.cpp/kingjones STRIX - halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next w4b, as a rootless Quadlet container on port 7770, plus the amd_iommu=off kernel change that retired the frank-npu endpoint. Repo commit 301914b; runbooks serverAppConfigs-docs/LLM/halogen-flash-server.md and dankclaw-iommu-translation-off.md.)_
+- halogen-flash-server (github.com/peonist-ai/halogen-flash-server) is an OCI-image-only inference server for gfx1151 Strix Halo, evaluated 2026-09-11 as a replacement for the llama.cpp frank-large unit on Dankclaw. Verified facts: - Image ghcr.io/peonist-ai/halogen-flash-server:0.5.8 (latest CHANGELOG tag). Ports: OpenAI-compatible **8731** (publish this), engine **8730** (NO auth — keep unpublished/loopback only). - deploy/entrypoint.sh modes: all (default; both roles in one container), engine, api (split so front-end restarts don't reload the model), bench [drafters] [max_tokens] [effort] [reps]. Engine binary is flash_serve, CLI --ck FILE (not --checkpoint), --slots/--ctx/--max-tok. - Weights: HF peonist-ai/halogen-qwen3.8-flash-next (public, ungated) — qwen38-flash-next-w4b.hgn 115.55 GiB + ck.overlay.hgn 2.31 GiB quality sidecar (auto-picked up; engine warns if absent) + tokenizer/ (8 files). Optional vision.hgn 0.84 GiB, overlay-speed.hgn (trades o_proj calibration). Container-side download via HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next. - Run requirements: --device /dev/kfd --device /dev/dri --group-add keep-groups --security-opt seccomp=unconfined --ipc=host, memlock unlimited, native Linux only (WSL2//dev/dxg unsupported), gfx1151 only (hard-rejected else). Reference SKU = Ryzen AI Max+ 395 / 128 GB — exactly Dankclaw, so the baked HALOGEN_MATMUL_TUNING_FILE does *not* warn. - Key flags (env-only, no config file, read once at startup): HALOGEN_CHECKPOINT (default /models/qwen38-flash-next-w4b.hgn), HALOGEN_CK_OVERLAY, HALOGEN_MODEL_ID (default halogen-qwen3.8-flash-next), HALOGEN_DRAFTER_DEFAULT=1 (MTP speculation on by default, byte-identical at temp 0), HALOGEN_PROMPT_CACHE=2 (numeric; 1 = byte-identical resume), HALOGEN_CACHE_ENTRIES=8, HALOGEN_MAX_TOKENS_CAP=65536, HALOGEN_CTX=262144, HALOGEN_KV_POOL_POSITIONS (default 524288 = 35.0 GB; 262144 = 27.8 GB), HALOGEN_KV_SLOTS=4, HALOGEN_KV_POOL_FIT=1, HALOGEN_HOST_RESERVE_GIB=20, HALOGEN_MAX_TOK= _(context: Research phase of a grilling session: replacing both the model and runtime of frank-large (Dankclaw) with halogen-flash-server, requested by Frank 2026-09-11. All facts read from the upstream repo (README, docs/FLAGS.md, deploy/entrypoint.sh, docker-compose.yml, CHANGELOG), the HF weights repo API, the ghcr image manifest, and live probes of Dankclaw.)_
 - On Dankclaw's Hermes stack, hermes-webui ran with the image-baked HERMES_HOME=/opt/data while the gateway used /data; /opt/data is an auto-created anonymous volume at the image's VOLUME point, so every container recreation (restart/redeploy) reset the webui's config home to the image-default template (provider: auto → openrouter.ai with no API key) — this is why model renames "didn't propagate" and chat failed with connect errors. Fix (2026-09-05, commit 184df9a): Environment=HERMES_HOME=/data in the hermes-webui quadlet so webui + gateway share one config/state root. Diagnostic order for "can't access models" on hermes: (1) check which config home each runtime actually reads (podman exec c printenv HERMES_HOME), (2) resolve provider→endpoint with the app's own code (hermes_cli.providers.resolve_user_provider(name, cfg["providers"]) accepts api/url/base_url keys on providers: dict entries), (3) probe the served endpoint with the exact model id. Model ids must match served aliases exactly (frank-large @192.168.0.143:7770, frank-small @192.168.0.235:7770, gemma4-it:e4b @192.168.0.143:7771 — FLM routes only exact registry tags).
 - Hermes (Dankclaw) model routing as of 2026-09-05: main default = frank-large (Dankclaw 192.168.0.143:7770/v1), small/aux routes = frank-small (Dankserver 192.168.0.235:7770/v1), vision + title_generation route to the exact FLM registry tag gemma4-it:e4b (192.168.0.143:7771/v1). The pre-rename aliases frank-default-large/-small/-sightsound are dead since the 2026-09-03 rename cutover — never route on them. Live config lives at /home/dank/Containers/hermes/data/config.yaml (owned by uid 109999; needs sudo to read/edit); the repo's dankclaw/hermes/config.yaml.example is a sanitized backup and can lag live, so diff live first before editing either. Signal adapter is currently disabled in live config (platforms.signal.enabled: false).
 - On the guarded franksdank.cloud vhosts (Caddy → Anubis :8923 → Authentik outpost), every unauthenticated request — roots and deep paths alike — is answered with HTTP **200** and an HTML challenge page (titleMaking sure you're not a bot!/title), not a 4xx. Verifying guard health by status code alone is meaningless; assert on leakage markers instead (copyparty UI title, homepage id="grid", services.json "category" JSON shape). Non-JS clients (curl/wget/WebDAV) can never solve the PoW challenge, so they are permanently parked at the challenge page — this is the entire security boundary for copyparty, which serves authenticated traffic with anonymous read perms (-v /w::r:ra,dank). Pinned by tests/e2e/specs/15-unauth-fetch.spec.ts (guest project, empty storage state): any guard-route regression that lets app bytes answer fails loudly; any gate implementation passes.