All ~173 managed skills read in full and merged, one polished reference page per skill-cluster: - LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split Triage, FreeToken Builds + deltas folded into the retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline, FLM pack registration, ik hybrid envelope, frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch split section, bench-window D-state/stale-read notes, second-endpoint bring-up). - Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State Triage. - Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit and Posture, SSH and Remote Access. - Infrastructure: Observability Provisioning + Homepage Status Probes; updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page preserved and extended). - FrankBook-Pro: ARR-Stack filled (API split, schema-first probing, auth-posture, auth-config trap); qBittorrent E2E verification. - Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and Edits, Command Center Dotfiles, PICO-8 Asset Standards. - Indexes: LLM Cross-Cutting References section, Operations/ and Agent/ index pages, root-index Skill Distillation section, AGENTS.md pointer. - log.md carries the wave plan. check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.
14 KiB
type, description, tags, timestamp
| type | description | tags | timestamp | ||
|---|---|---|---|---|---|
| reference | Hermes is the self-improving AI agent from Nous Research. |
|
2026-09-15 |
Hermes
Hermes is the self-improving AI agent from Nous Research. It runs in a terminal, Telegram, Discord, Slack, WhatsApp, or Signal, and keeps a persistent learning loop: it creates skills from experience, searches past conversations, and builds a model of the user across sessions.
Host: Dankclaw — the gateway itself publishes no port (it runs in the
host network namespace and dials the Signal daemon at 127.0.0.1:8080); the
user-facing web surface is Hermes-WebUI on :8787, reached via
hermes.franksdank.cloud — internal-only by DNS absence (no public Porkbun A
record) — Docs: https://hermes-agent.nousresearch.com/docs
Persistent state: /home/dank/Containers/hermes/data on Dankclaw, bind-mounted
at /data in all three Hermes containers (gateway, webui, dashboard). Each
container additionally gets an anonymous volume at /opt/data, which is
re-created empty on every container recreation — anything that lives only there
is not durable. The tracked, sanitized configuration baseline is
dankclaw/hermes/config.yaml.example.
Facts (from research, primary source)
| Item | Value |
|---|---|
| Image | built from NousResearch/hermes-agent (official Dockerfile in repo) |
| Upstream latest | agent v0.21.4 / CalVer v2026.9.21 (2026-09-21); hermes-webui v0.52.113 (2026-08-14). Pin the CalVer tag, not :latest. |
| License | MIT |
| Model backends | Nous Portal, OpenRouter, OpenAI, or any custom endpoint |
| Interfaces | TUI, Telegram, Discord, Slack, WhatsApp, Signal |
| Memory | FTS5 session search, Honcho user modeling |
| Skills | self-created, self-improving during use |
Configuration
- Point Hermes at the local model endpoint or an external provider through its provider config.
- The gateway runs
hermes gateway run --force -vin the host network namespace (Network=hostin its Quadlet unit) and reaches the Signal daemon athttp://127.0.0.1:8080. That loopback hop works only because the gateway is in the host netns — see Container Network Namespaces. The daemon itself is a host user unit (signal-cli-daemon.service), not a container, and binds0.0.0.0:8080, so the Signal HTTP API is reachable from the whole LAN. - The data volume persists skills, memory, and session history.
- Hermes has two named local providers:
dankclaw-largereacheshttp://192.168.0.143:7770/v1and exposesfrank-large;dankserver-smallreacheshttp://192.168.0.235:7770/v1and exposesfrank-medium(renamed fromfrank-smallon 2026-09-15). - The live
config.yamlcontains secrets and user state and is not committed; the tracked example is the sanitized backup of its non-secret settings.
Volumes
| Container path | Host path (live) | Backing |
|---|---|---|
/data |
/home/dank/Containers/hermes/data |
bind mount (durable; shared by all three containers) |
/webui |
/home/dank/Containers/hermes/webui |
bind mount (webui only — the bootstrap.py venv lives here) |
/opt/data |
anonymous volume | image VOLUME; fresh empty volume per recreation |
Gotchas
- Hermes is under active development; pin the image tag for reproducibility.
- Model choice drives quality; the local
frank-large(Dankclaw, 7770) is the intended backend. - Do not expose the gateway without auth — put it behind Caddy with basic auth.
Browser automation (Chrome DevTools Protocol daemon)
The agent's browser tool is an MCP client (mcp_tool.py in the image) that
spawns npx -y chrome-devtools-mcp@latest --browserUrl=http://127.0.0.1:9222
per session; the browser itself lives in a dedicated daemon container, the
digest-pinned headless-shell unit chrome-devtools-protocol.container (host
network, loopback-only bind on 127.0.0.1:9222). Both consumers — the Signal
gateway and the webui server — run in the host network namespace, so they can
dial the daemon on loopback; nothing on the LAN can reach the CDP endpoint.
The wiring lives in the live /data/config.yaml under mcp_servers.chrome-devtools
(command npx, args -y chrome-devtools-mcp@latest --browserUrl=http://127.0.0.1:9222).
The npx package cache lives in the container layer, so a recreation (not just a
restart) pays one first-call npm download; subsequent calls are cached.
Config-home drift ("cannot connect" to models that answer curl)
podman exec <container> printenv HERMES_HOME for each hermes container. If
the webui reads the image default (/opt/data) while the gateway reads
/data, the webui runs the image-default template config (provider: auto →
openrouter.ai with no key = connect failure).
The recreation trap is the same one the volumes table shows: image VOLUME
points get a fresh anonymous volume on every container recreation
(podman run --replace --rm) — edits inside such a mount are lost on
restart/redeploy. Compare volume ids before/after a restart to confirm.
Fix: pin Environment=HERMES_HOME=/data in the quadlet unit (shared state
root), deploy service-scoped. Verify with the app's own code, not guesses:
from api import config as C; C.resolve_model_provider("<model-id>")
# -> (model_id, provider, base_url); bare ids map to the owning provider by
# scanning the providers: dict models allow-list; @provider:model forces one.
hermes_cli.providers.resolve_user_provider(name, cfg["providers"])
# -> ProviderDef with real base_url; accepts api/url/base_url keys.
Named providers (providers: dict) are a catalog + connection source;
custom:<slug> connections resolve only via custom_providers[] entries. If
neither holds the provider name, the request has no endpoint and fails to
connect. Probe the endpoint with the exact served model id — the retired
FLM/NPU endpoints routed only by exact registry tag and answered HTTP 200
with {"error":"Failed to load <tag> model!"} for unknown tags; status-code
checks miss this class entirely.
Live config edits
Live file: /home/dank/Containers/hermes/data/config.yaml (dir drwx------
uid 109999; all reads/edits need sudo). The repo file
dankclaw/hermes/config.yaml.example is a sanitized backup and can lag
live — always diff the live file first; never treat the example as truth.
- Backup first:
sudo cp -a …/config.yaml …/config.yaml.bak-YYYYMMDD(in-dir.AGENTTRASH/also exists for removals). - Apply with surgical
sudo sed -ior full-file write; YAML comments in the live file are tool-generated — preserve them. - Sweep stale references across the whole data dir:
sudo grep -rln "<old-id>" /home/dank/Containers/hermes/data --include="*.yaml" --include="*.json". Session files/MEMORY.md are historical — leave them. - Derived caches:
webui/models_cache.jsonis keyed by a config fingerprint (_source_fingerprint.config_yamlmtime/size); after edits it is stale. Let it regenerate lazily or trash it — never hand-edit. - Restart the model-consuming units (they parse config at startup):
systemctl --user restart hermes-webui.service hermes.service.hermes-dashboard.serviceis unaffected by routing changes. - Verify behaviorally: one tiny completion per endpoint with the exact
served id — POST
/v1/chat/completionsand read the body, never just the status code (endpoint names rotate; see the LLM knowledge base). - Sync the repo backup (
dankclaw/hermes/config.yaml.example) to the live shape, redacting secrets (password_hash, phone account stay live-only), then./tests/run.shbefore commit.
provider: custom + base_url entries route by base_url and send the model
id verbatim; named providers resolve via their api: URL. Renaming a
served alias anywhere (serve scripts, env ALIAS) MUST propagate to this
config in the same change or Hermes requests 404 on nonexistent models —
this exact gap caused the 2026-09-05 outage.
Auxiliary routing against local backends
When a Hermes feature that calls an auxiliary task (e.g. title generation) fails against a local endpoint, the usual cause is a request shape the local runtime does not implement, not a routing mistake.
Diagnose with the real call path — run the actual function inside the container with the app's own constants (the probe cannot drift from production):
base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \
'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \
podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \
podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py'
Copy the file AFTER any restart (--replace --rm discards the container
layer). The CLI at /opt/hermes/hermes uses system python and dies
No module named 'yaml'; use /opt/hermes/.venv/bin/python.
import sys; sys.path.insert(0, "/opt/hermes")
from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text
from agent.auxiliary_client import call_llm
resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT})
What actually breaks, and the fix:
-
response_format(json_object / json_schema) is a hard 400 on every local backend here — neither FreeToken nor halogen implements constrained decoding. Config cannot override it:call_llmmerges the task'sextra_bodyfirst and thenupdate()s the caller's, so the hardcoded one wins. Only temperature and max_tokens have reactive retry paths. -
Fix: move that one task to the Responses transport, which never forwards
response_format:auxiliary: title_generation: provider: dankserver-small # names track the live config; diff first model: frank-small api_mode: codex_responses # real per-task key extra_body: reasoning: effort: none # mandatory, see below -
Disable thinking for any cap-limited task. The adapter omits reasoning only when config says
reasoning.enabled: false. Thereasoning_effort: noneshorthand produces exactly that and leaves the server default (thinking ON) — the 64-token title budget gets spent on the think: empty content,finish_reason: length. The explicitextra_body.reasoning.effort: noneinstead sends{"effort":"none","summary":"auto"}, which the local/v1/responseshonors. -
Both local runtimes also answer
/v1/responsesand/v1/messages— no proxy needed for the alternate transport.
Verify and land: re-run the probe with no api_mode/reasoning kwargs —
it must pass from config alone (that is the production path); keep the live
file and the repo example in step; ./tests/run.sh, commit, push — a Hermes
config change is docs+example only, no service deploy needed.
Gotchas: auxiliary.<task>.enabled: false is checked BEFORE
apply_instant_title, so disabling a broken title task also removes the
derived title — it does not degrade gracefully the way a failure does. A
local call can also fail at the model level: a 200 with an error-text body,
or a chatty non-title answer. Always read the raw content, not the status.
Webui password rotation ("Hermex")
Auth contract (verified live 2026-08-31): POST /api/auth/login with JSON
{"password":"…"} → 200 + signed HttpOnly hermes_session cookie (24 h TTL)
on match, 401 on miss. No username field. /health answers 200
unauthenticated and / answers 302 — never verify auth via those; probe
login with a deliberately wrong password (expect 401) and the real one
(expect 200). Details on the webui surface: Hermes-WebUI.
- Run the whole probe as a heredoc'd bash script
(
ssh -o BatchMode=yes dankclaw 'bash -s') — the host's zsh mangles inline${…}even inside single-quoted ssh args; never interpolate curl-w %{http_code}inline. - Update BOTH env files with
sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=<new>/":/home/dank/serverAppConfigs/.env([Service] EnvironmentFile)/home/dank/serverAppConfigs/dankclaw/.env— the effective one: the quadlet's[Container] EnvironmentFile=is what the container reads at create; rotating only the root file leaves the running container serving the old secret.
systemctl --user restart hermes-webui.service(unit recreation re-reads env; no deploy needed if the unit file itself didn't change).- Verify wrong-pw → 401, right-pw → 200, from host loopback and a LAN
client at
http://192.168.0.143:8787(webui publishes on the LAN interface since commit4443897; off-network ingress is SSH tunnel only — usertunnel, port 22, key-only auth, forward remote end localhost:8787).
Gotchas: sudo podman exec fails exit 125 here — root's namespace does not
see rootless containers; run podman exec hermes-webui printenv HERMES_WEBUI_PASSWORD unprivileged (the ssh user owns the env files, mode
600). If rotating because a secret leaked into transcripts/logs, rotate in
place via a script that fetches the live value from the running container
(printenv) instead of transcribing it.
See Also
- Hermes-Dashboard — the LAN browser management UI
- Hermes-WebUI — the Hermex server (phone-facing API, SSH-tunneled)
- Dankclaw Stack
- OpenWebUI — the chat UI that pairs with Hermes
- Knowledge base: SearXNG (Hermes as the search agent)
- Container Network Namespaces — why the gateway and webui can dial loopback services and the dashboard cannot