docs: distill the managed skill library into the vault (waves 1-7)
All ~173 managed skills read in full and merged, one polished reference page per skill-cluster: - LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split Triage, FreeToken Builds + deltas folded into the retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline, FLM pack registration, ik hybrid envelope, frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch split section, bench-window D-state/stale-read notes, second-endpoint bring-up). - Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State Triage. - Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit and Posture, SSH and Remote Access. - Infrastructure: Observability Provisioning + Homepage Status Probes; updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page preserved and extended). - FrankBook-Pro: ARR-Stack filled (API split, schema-first probing, auth-posture, auth-config trap); qBittorrent E2E verification. - Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and Edits, Command Center Dotfiles, PICO-8 Asset Standards. - Indexes: LLM Cross-Cutting References section, Operations/ and Agent/ index pages, root-index Skill Distillation section, AGENTS.md pointer. - log.md carries the wave plan. check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.
This commit is contained in:
Frank
@@ -81,6 +81,178 @@ The wiring lives in the live `/data/config.yaml` under `mcp_servers.chrome-devto
|
||||
The npx package cache lives in the container layer, so a recreation (not just a
|
||||
restart) pays one first-call npm download; subsequent calls are cached.
|
||||
|
||||
## Config-home drift ("cannot connect" to models that answer curl)
|
||||
|
||||
`podman exec <container> printenv HERMES_HOME` for each hermes container. If
|
||||
the webui reads the image default (`/opt/data`) while the gateway reads
|
||||
`/data`, the webui runs the image-default template config (`provider: auto` →
|
||||
openrouter.ai with no key = connect failure).
|
||||
|
||||
The recreation trap is the same one the volumes table shows: image `VOLUME`
|
||||
points get a fresh anonymous volume on every container recreation
|
||||
(`podman run --replace --rm`) — edits inside such a mount are lost on
|
||||
restart/redeploy. Compare volume ids before/after a restart to confirm.
|
||||
|
||||
Fix: pin `Environment=HERMES_HOME=/data` in the quadlet unit (shared state
|
||||
root), deploy service-scoped. Verify with the app's own code, not guesses:
|
||||
|
||||
```python
|
||||
from api import config as C; C.resolve_model_provider("<model-id>")
|
||||
# -> (model_id, provider, base_url); bare ids map to the owning provider by
|
||||
# scanning the providers: dict models allow-list; @provider:model forces one.
|
||||
hermes_cli.providers.resolve_user_provider(name, cfg["providers"])
|
||||
# -> ProviderDef with real base_url; accepts api/url/base_url keys.
|
||||
```
|
||||
|
||||
Named providers (`providers:` dict) are a catalog + connection source;
|
||||
`custom:<slug>` connections resolve only via `custom_providers[]` entries. If
|
||||
neither holds the provider name, the request has no endpoint and fails to
|
||||
connect. Probe the endpoint with the **exact served model id** — the retired
|
||||
FLM/NPU endpoints routed only by exact registry tag and answered HTTP 200
|
||||
with `{"error":"Failed to load <tag> model!"}` for unknown tags; status-code
|
||||
checks miss this class entirely.
|
||||
|
||||
## Live config edits
|
||||
|
||||
Live file: `/home/dank/Containers/hermes/data/config.yaml` (dir `drwx------`
|
||||
uid 109999; all reads/edits need `sudo`). The repo file
|
||||
`dankclaw/hermes/config.yaml.example` is a sanitized backup and **can lag
|
||||
live — always diff the live file first; never treat the example as truth**.
|
||||
|
||||
1. **Backup first:** `sudo cp -a …/config.yaml …/config.yaml.bak-YYYYMMDD`
|
||||
(in-dir `.AGENTTRASH/` also exists for removals).
|
||||
2. **Apply** with surgical `sudo sed -i` or full-file write; YAML comments in
|
||||
the live file are tool-generated — preserve them.
|
||||
3. **Sweep stale references** across the whole data dir:
|
||||
`sudo grep -rln "<old-id>" /home/dank/Containers/hermes/data
|
||||
--include="*.yaml" --include="*.json"`. Session files/MEMORY.md are
|
||||
historical — leave them.
|
||||
4. **Derived caches:** `webui/models_cache.json` is keyed by a config
|
||||
fingerprint (`_source_fingerprint.config_yaml` mtime/size); after edits it
|
||||
is stale. Let it regenerate lazily or trash it — never hand-edit.
|
||||
5. **Restart the model-consuming units** (they parse config at startup):
|
||||
`systemctl --user restart hermes-webui.service hermes.service`.
|
||||
`hermes-dashboard.service` is unaffected by routing changes.
|
||||
6. **Verify behaviorally**: one tiny completion per endpoint with the exact
|
||||
served id — POST `/v1/chat/completions` and read the body, never just the
|
||||
status code (endpoint names rotate; see the LLM knowledge base).
|
||||
7. **Sync the repo backup** (`dankclaw/hermes/config.yaml.example`) to the
|
||||
live shape, redacting secrets (`password_hash`, phone account stay
|
||||
live-only), then `./tests/run.sh` before commit.
|
||||
|
||||
`provider: custom` + `base_url` entries route by base_url and send the model
|
||||
id verbatim; named providers resolve via their `api:` URL. **Renaming a
|
||||
served alias anywhere (serve scripts, env ALIAS) MUST propagate to this
|
||||
config in the same change** or Hermes requests 404 on nonexistent models —
|
||||
this exact gap caused the 2026-09-05 outage.
|
||||
|
||||
## Auxiliary routing against local backends
|
||||
|
||||
When a Hermes feature that calls an auxiliary task (e.g. title generation)
|
||||
fails against a local endpoint, the usual cause is a **request shape the
|
||||
local runtime does not implement**, not a routing mistake.
|
||||
|
||||
Diagnose with the real call path — run the actual function inside the
|
||||
container with the app's own constants (the probe cannot drift from
|
||||
production):
|
||||
|
||||
```bash
|
||||
base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \
|
||||
'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \
|
||||
podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \
|
||||
podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py'
|
||||
```
|
||||
|
||||
Copy the file AFTER any restart (`--replace --rm` discards the container
|
||||
layer). The CLI at `/opt/hermes/hermes` uses system python and dies
|
||||
`No module named 'yaml'`; use `/opt/hermes/.venv/bin/python`.
|
||||
|
||||
```python
|
||||
import sys; sys.path.insert(0, "/opt/hermes")
|
||||
from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text
|
||||
from agent.auxiliary_client import call_llm
|
||||
resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
|
||||
temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT})
|
||||
```
|
||||
|
||||
What actually breaks, and the fix:
|
||||
|
||||
- **`response_format` (json_object / json_schema) is a hard 400 on every
|
||||
local backend here** — neither FreeToken nor halogen implements constrained
|
||||
decoding. Config cannot override it: `call_llm` merges the task's
|
||||
`extra_body` first and then `update()`s the caller's, so the hardcoded one
|
||||
wins. Only temperature and max_tokens have reactive retry paths.
|
||||
- **Fix: move that one task to the Responses transport**, which never
|
||||
forwards `response_format`:
|
||||
|
||||
```yaml
|
||||
auxiliary:
|
||||
title_generation:
|
||||
provider: dankserver-small # names track the live config; diff first
|
||||
model: frank-small
|
||||
api_mode: codex_responses # real per-task key
|
||||
extra_body:
|
||||
reasoning:
|
||||
effort: none # mandatory, see below
|
||||
```
|
||||
|
||||
- **Disable thinking for any cap-limited task.** The adapter omits reasoning
|
||||
only when config says `reasoning.enabled: false`. The
|
||||
`reasoning_effort: none` *shorthand* produces exactly that and leaves the
|
||||
server default (thinking ON) — the 64-token title budget gets spent on the
|
||||
think: empty content, `finish_reason: length`. The explicit
|
||||
`extra_body.reasoning.effort: none` instead sends
|
||||
`{"effort":"none","summary":"auto"}`, which the local `/v1/responses`
|
||||
honors.
|
||||
- Both local runtimes also answer `/v1/responses` and `/v1/messages` — no
|
||||
proxy needed for the alternate transport.
|
||||
|
||||
Verify and land: re-run the probe with **no** `api_mode`/reasoning kwargs —
|
||||
it must pass from config alone (that is the production path); keep the live
|
||||
file and the repo example in step; `./tests/run.sh`, commit, push — a Hermes
|
||||
config change is docs+example only, no service deploy needed.
|
||||
|
||||
Gotchas: `auxiliary.<task>.enabled: false` is checked BEFORE
|
||||
`apply_instant_title`, so disabling a broken title task also removes the
|
||||
derived title — it does not degrade gracefully the way a failure does. A
|
||||
local call can also fail at the model level: a 200 with an error-text body,
|
||||
or a chatty non-title answer. Always read the raw content, not the status.
|
||||
|
||||
## Webui password rotation ("Hermex")
|
||||
|
||||
Auth contract (verified live 2026-08-31): `POST /api/auth/login` with JSON
|
||||
`{"password":"…"}` → 200 + signed HttpOnly `hermes_session` cookie (24 h TTL)
|
||||
on match, 401 on miss. No username field. `/health` answers 200
|
||||
unauthenticated and `/` answers 302 — **never verify auth via those**; probe
|
||||
login with a deliberately wrong password (expect 401) and the real one
|
||||
(expect 200). Details on the webui surface: [[Hermes-WebUI]].
|
||||
|
||||
1. Run the whole probe as a heredoc'd bash script
|
||||
(`ssh -o BatchMode=yes dankclaw 'bash -s'`) — the host's zsh mangles
|
||||
inline `${…}` even inside single-quoted ssh args; never interpolate curl
|
||||
`-w %{http_code}` inline.
|
||||
2. Update BOTH env files with
|
||||
`sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=<new>/"`:
|
||||
- `/home/dank/serverAppConfigs/.env` ([Service] EnvironmentFile)
|
||||
- `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one**: the
|
||||
quadlet's `[Container] EnvironmentFile=` is what the container reads at
|
||||
create; rotating only the root file leaves the running container serving
|
||||
the old secret.
|
||||
3. `systemctl --user restart hermes-webui.service` (unit recreation re-reads
|
||||
env; no deploy needed if the unit file itself didn't change).
|
||||
4. Verify wrong-pw → 401, right-pw → 200, from host loopback **and** a LAN
|
||||
client at `http://192.168.0.143:8787` (webui publishes on the LAN
|
||||
interface since commit `4443897`; off-network ingress is SSH tunnel only —
|
||||
user `tunnel`, port **22**, key-only auth, forward remote end
|
||||
localhost:8787).
|
||||
|
||||
Gotchas: `sudo podman exec` fails exit 125 here — root's namespace does not
|
||||
see rootless containers; run `podman exec hermes-webui printenv
|
||||
HERMES_WEBUI_PASSWORD` unprivileged (the ssh user owns the env files, mode
|
||||
600). If rotating because a secret leaked into transcripts/logs, rotate in
|
||||
place via a script that fetches the live value from the running container
|
||||
(`printenv`) instead of transcribing it.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Hermes-Dashboard]] — the LAN browser management UI
|
||||
|
||||
Reference in New Issue
Block a user