Files
serverAppConfigs/serverAppConfigs-docs/Dankclaw/Hermes.md
T
Frank e7f0d93c86 docs(vault): fleet-wide fact re-verification pass (67 pages)
The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.

Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.
2026-10-18 13:15:43 -07:00

19 KiB

type, description, tags, timestamp
type description tags timestamp
reference Hermes is the self-improving AI agent from Nous Research.
dankclaw
hermes
2026-09-15

Hermes

Hermes is the self-improving AI agent from Nous Research. It runs in a terminal, Telegram, Discord, Slack, WhatsApp, or Signal, and keeps a persistent learning loop: it creates skills from experience, searches past conversations, and builds a model of the user across sessions.

Host: Dankclaw — the gateway itself publishes no port (it runs in the host network namespace; podman inspect systemd-hermes → net=host, no port bindings, 2026-10-18). Note the container name drift: the unit is hermes.service but the generated start line is podman run --name systemd-%N … (systemctl --user cat hermes.service), so podman ps shows the gateway as systemd-hermes — the repo unit sets no ContainerName= key even though its header comment says "container name: hermes". The user-facing web surface is Hermes-WebUI on :8787, reached via hermes.franksdank.cloud — internal-only by DNS absence (no public Porkbun A record; the name resolves from the Pi-hole hosts file, dankclaw/pihole/addn-hosts:39) — Docs: https://hermes-agent.nousresearch.com/docs

Signal is not carrying traffic today (measured 2026-10-18). The gateway boots and stays up, but its own log says Platform 'signal' is explicitly disabled by platforms.signal.enabled: false in config.yaml, so the credentials found in the environment (SIGNAL_HTTP_URL) will NOT start its adapter (journalctl --user -u hermes 2026-10-18), and the host daemon it would dial is not running either: systemctl --user is-enabled signal-cli-daemon → disabled, is-active → inactive, and nothing listens on :8080 (ss -ltn). Everything below still describes the designed wiring; the remedy for both halves is in the log line (platforms.signal.enabled: true) plus enabling the daemon unit.

Persistent state: /home/dank/Containers/hermes/data on Dankclaw, bind-mounted at /data in all three Hermes containers (gateway, webui, dashboard — hermes.container:36, hermes-webui.container:46, hermes-dashboard.container:24). Each container additionally gets an anonymous volume at /opt/data, which is re-created empty on every container recreation — anything that lives only there is not durable. The tracked, sanitized configuration baseline is dankclaw/hermes/config.yaml.example.

Facts (from research, primary source)

Item Value
Image localhost/hermes-agent:latest, built from NousResearch/hermes-agent (official Dockerfile in repo); Pull=never in all three units (hermes.container:28-29, hermes-webui.container:35-36, hermes-dashboard.container:19-20), so the tag never moves at runtime. Live image: built 2026-09-24, OCI revision label 749220ef0007f8d87bd1531f1c24b0fe93816385 (podman image inspect localhost/hermes-agent:latest 2026-10-18) — that build is what is running.
Upstream latest agent v0.21.4 / CalVer v2026.9.21 (2026-09-21); hermes-webui v0.52.113 (2026-08-14). Pin the CalVer tag, not :latest. (unverified 2026-10-18: the local image records only a git revision, and the webui version endpoint answers 401 without a session, so neither number can be re-derived from the host.)
License MIT
Model backends Nous Portal, OpenRouter, OpenAI, or any custom endpoint
Interfaces TUI, Telegram, Discord, Slack, WhatsApp, Signal
Memory FTS5 session search, Honcho user modeling
Skills self-created, self-improving during use

Configuration

  • Point Hermes at the local model endpoint or an external provider through its provider config.
  • The gateway runs hermes gateway run --force -v in the host network namespace (Exec= + Network=host, dankclaw/hermes/quadlet/hermes.container:27,37) and reaches the Signal daemon at http://127.0.0.1:8080 (SIGNAL_HTTP_URL, hermes.container:33). That loopback hop works only because the gateway is in the host netns — see Container Network Namespaces. The daemon itself is a host user unit (~/.config/systemd/user/signal-cli-daemon.service), not a container. Today it is disabled + inactive and nothing is bound to :8080 (systemctl --user is-enabled/is-active, ss -ltn, both 2026-10-18), and the gateway's own config gate (platforms.signal.enabled: false) keeps the adapter off even if the daemon came back — see the dated note at the top of this page. When the daemon does run it binds 0.0.0.0:8080, i.e. the Signal HTTP API is reachable from the whole LAN.
  • The data volume persists skills, memory, and session history.
  • Hermes has two named local providers (the dankclaw-large name is confirmed live from the gateway log — resolve_provider_client: named custom provider 'dankclaw-large' has no resolvable api_key, journalctl --user -u hermes 2026-10-18; the base URLs below are the repo record, since /data/config.yaml is not readable without sudo): dankclaw-large reaches http://192.168.0.143:7770/v1 and exposes frank-dragoon — verified from the host with curl http://127.0.0.1:7770/v1/models → single id frank-dragoon, max_model_len 200704 (2026-10-18; write 127.0.0.1, not localhost, because that port is a rootless pasta publish and is IPv4-only); dankserver-small reaches http://192.168.0.235:7770/v1 and, as of 2026-10-18, that endpoint serves only frank-archon (GET http://192.168.0.235:7770/v1/models → [{"id":"frank-archon"}]). The name this page used to carry for it — frank-medium, renamed from frank-small on 2026-09-15 — is retired-era: nothing on that endpoint answers to it today, so if the live config.yaml still names frank-medium the provider is dead weight and requests will 404 on a nonexistent model. Diff the live config before trusting either name.
  • The live config.yaml contains secrets and user state and is not committed; the tracked example is the sanitized backup of its non-secret settings.

Volumes

Container path Host path (live) Backing
/data /home/dank/Containers/hermes/data bind mount (durable; shared by all three containers)
/webui /home/dank/Containers/hermes/webui bind mount (webui only — the bootstrap.py venv lives here)
/opt/data anonymous volume image VOLUME; fresh empty volume per recreation

Gotchas

  • Hermes is under active development; pin the image tag for reproducibility. The units already pin by absence of pulling (Pull=never) and by building localhost/hermes-agent:latest locally — a rebuild, not a pull, is what changes the code.
  • Model choice drives quality; the local frank-dragoon (Dankclaw, 7770) is the intended backend.
  • Do not expose the gateway without auth — put it behind Caddy with basic auth. The gateway itself publishes no port today; the surfaces that are exposed are the webui (hermes. vhost, exempt by construction because the app enforces its own password — see Hermes-WebUI) and the dashboard (hermesadmin. vhost → 192.168.0.143:9119, app-enforced basic auth — see Hermes-Dashboard).

Browser automation (Chrome DevTools Protocol daemon)

The agent's browser tool is an MCP client (mcp_tool.py in the image) that spawns npx -y chrome-devtools-mcp@latest --browserUrl=http://127.0.0.1:9222 per session; the browser itself lives in a dedicated daemon container, the digest-pinned headless-shell unit chrome-devtools-protocol.container (host network, loopback-only bind on 127.0.0.1:9222 — ss -ltnp 2026-10-18 shows 127.0.0.1:9222 headless-shell and nothing on 0.0.0.0; image docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5…, dankclaw/hermes/quadlet/chrome-devtools-protocol.container:28). Both consumers — the Signal gateway and the webui server — run in the host network namespace (podman inspect → net=host for both, 2026-10-18), so they can dial the daemon on loopback; nothing on the LAN can reach the CDP endpoint. GET http://127.0.0.1:9222/json/version answers 200 today with Browser: Chrome/151.0.7922.109, Protocol-Version: 1.3 (2026-10-18). The wiring lives in the live /data/config.yaml under mcp_servers.chrome-devtools (command npx, args -y chrome-devtools-mcp@latest --browserUrl=http://127.0.0.1:9222) — that file is mode drwx------ uid 109999, so reading it needs sudo (a non-privileged read is EACCES, checked 2026-10-18). The npx package cache lives in the container layer, so a recreation (not just a restart) pays one first-call npm download; subsequent calls are cached.

Config-home drift ("cannot connect" to models that answer curl)

podman exec <container> printenv HERMES_HOME for each hermes container. If the webui reads the image default (/opt/data) while the gateway reads /data, the webui runs the image-default template config (provider: auto → openrouter.ai with no key = connect failure).

The recreation trap is the same one the volumes table shows: image VOLUME points get a fresh anonymous volume on every container recreation (podman run --replace --rm) — edits inside such a mount are lost on restart/redeploy. Compare volume ids before/after a restart to confirm.

Fix: pin Environment=HERMES_HOME=/data in the quadlet unit (shared state root), deploy service-scoped. Verify with the app's own code, not guesses:

from api import config as C; C.resolve_model_provider("<model-id>")
# -> (model_id, provider, base_url); bare ids map to the owning provider by
#    scanning the providers: dict models allow-list; @provider:model forces one.
hermes_cli.providers.resolve_user_provider(name, cfg["providers"])
# -> ProviderDef with real base_url; accepts api/url/base_url keys.

Named providers (providers: dict) are a catalog + connection source; custom:<slug> connections resolve only via custom_providers[] entries. If neither holds the provider name, the request has no endpoint and fails to connect. Probe the endpoint with the exact served model id — the retired FLM/NPU endpoints routed only by exact registry tag and answered HTTP 200 with {"error":"Failed to load <tag> model!"} for unknown tags; status-code checks miss this class entirely.

Live config edits

Live file: /home/dank/Containers/hermes/data/config.yaml (dir drwx------ uid 109999 — ls -ld /home/dank/Containers/hermes/data → drwx------ 29 109999 109999 2026-10-18; a non-privileged read returns EACCES, so all reads/edits need sudo). The repo file dankclaw/hermes/config.yaml.example is a sanitized backup and can lag live — always diff the live file first; never treat the example as truth.

  1. Backup first: sudo cp -a …/config.yaml …/config.yaml.bak-YYYYMMDD (in-dir .AGENTTRASH/ also exists for removals (unverified 2026-10-18: inside the 0700 data dir, unreadable without sudo)).
  2. Apply with surgical sudo sed -i or full-file write; YAML comments in the live file are tool-generated — preserve them.
  3. Sweep stale references across the whole data dir: sudo grep -rln "<old-id>" /home/dank/Containers/hermes/data --include="*.yaml" --include="*.json". Session files/MEMORY.md are historical — leave them.
  4. Derived caches: webui/models_cache.json is keyed by a config fingerprint (_source_fingerprint.config_yaml mtime/size); after edits it is stale. Let it regenerate lazily or trash it — never hand-edit.
  5. Restart the model-consuming units (they parse config at startup): systemctl --user restart hermes-webui.service hermes.service. hermes-dashboard.service is unaffected by routing changes.
  6. Verify behaviorally: one tiny completion per endpoint with the exact served id — POST /v1/chat/completions and read the body, never just the status code (endpoint names rotate; see the LLM knowledge base).
  7. Sync the repo backup (dankclaw/hermes/config.yaml.example) to the live shape, redacting secrets (password_hash, phone account stay live-only), then ./tests/run.sh before commit.

provider: custom + base_url entries route by base_url and send the model id verbatim; named providers resolve via their api: URL. Renaming a served alias anywhere (serve scripts, env ALIAS) MUST propagate to this config in the same change or Hermes requests 404 on nonexistent models — this exact gap caused the 2026-09-05 outage.

Auxiliary routing against local backends

When a Hermes feature that calls an auxiliary task (e.g. title generation) fails against a local endpoint, the usual cause is a request shape the local runtime does not implement, not a routing mistake.

Diagnose with the real call path — run the actual function inside the container with the app's own constants (the probe cannot drift from production):

base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \
  'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \
   podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \
   podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py'

Copy the file AFTER any restart (--replace --rm discards the container layer). The CLI at /opt/hermes/hermes uses system python and dies No module named 'yaml'; use /opt/hermes/.venv/bin/python.

import sys; sys.path.insert(0, "/opt/hermes")
from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text
from agent.auxiliary_client import call_llm
resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
                temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT})

What actually breaks, and the fix:

  • response_format (json_object / json_schema) is a hard 400 on every local backend here — neither FreeToken nor halogen implements constrained decoding. Config cannot override it: call_llm merges the task's extra_body first and then update()s the caller's, so the hardcoded one wins. Only temperature and max_tokens have reactive retry paths.

  • Fix: move that one task to the Responses transport, which never forwards response_format:

    auxiliary:
      title_generation:
        provider: dankserver-small        # names track the live config; diff first
        model: frank-zealot
        api_mode: codex_responses        # real per-task key
        extra_body:
          reasoning:
            effort: none                 # mandatory, see below
    

    Before copying this: as of 2026-10-18 frank-zealot is served on Dankclaw :7771, and http://192.168.0.235:7770/v1 — the endpoint this provider name is recorded against — answers only frank-archon. The provider/model pairing above is the shape that was landed in 2026; re-derive it from the live config before reusing it.

  • Disable thinking for any cap-limited task. The adapter omits reasoning only when config says reasoning.enabled: false. The reasoning_effort: none shorthand produces exactly that and leaves the server default (thinking ON) — the 64-token title budget gets spent on the think: empty content, finish_reason: length. The explicit extra_body.reasoning.effort: none instead sends {"effort":"none","summary":"auto"}, which the local /v1/responses honors.

  • Both local runtimes also answer /v1/responses and /v1/messages — no proxy needed for the alternate transport.

Verify and land: re-run the probe with no api_mode/reasoning kwargs — it must pass from config alone (that is the production path); keep the live file and the repo example in step; ./tests/run.sh, commit, push — a Hermes config change is docs+example only, no service deploy needed.

Gotchas: auxiliary.<task>.enabled: false is checked BEFORE apply_instant_title, so disabling a broken title task also removes the derived title — it does not degrade gracefully the way a failure does. A local call can also fail at the model level: a 200 with an error-text body, or a chatty non-title answer. Always read the raw content, not the status.

Webui password rotation ("Hermex")

Auth contract (verified live 2026-08-31, re-checked 2026-10-18): POST /api/auth/login with JSON {"password":"…"} → 200 + signed HttpOnly hermes_session cookie (24 h TTL) on match, 401 on miss. No username field. /health answers 200 unauthenticated (curl http://127.0.0.1:8787/health → 200 with {"status":"ok", …} today) and / answers 302 (curl -o /dev/null -w %{http_code} http://127.0.0.1:8787/ → 302 today) — never verify auth via those; probe login with a deliberately wrong password (expect 401) and the real one (expect 200). Details on the webui surface: Hermes-WebUI.

  1. Run the whole probe as a heredoc'd bash script (ssh -o BatchMode=yes dankclaw 'bash -s') — the host's zsh mangles inline ${…} even inside single-quoted ssh args; never interpolate curl -w %{http_code} inline.
  2. Update BOTH env files with sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=<new>/":
    • /home/dank/serverAppConfigs/.env — the repo-root checkout's env file (exists, -rw------- dank dank, 2026-10-18) but not loaded by this unit: keep it in sync only so other consumers do not drift.
    • /home/dank/serverAppConfigs/dankclaw/.env — the effective one, for both [Container] and [Service]: systemctl --user show hermes-webui -p EnvironmentFiles reports exactly /home/dank/serverAppConfigs/dankclaw/.env (2026-10-18; dankclaw/hermes/quadlet/hermes-webui.container:38,62). That is what the container reads at create; rotating only the root file leaves the running container serving the old secret.
  3. systemctl --user restart hermes-webui.service (unit recreation re-reads env; no deploy needed if the unit file itself didn't change).
  4. Verify wrong-pw → 401, right-pw → 200, from host loopback and a LAN client at http://192.168.0.143:8787 (webui publishes on the LAN interface since commit 4443897, 2026-08-31 "publish webui on LAN interface instead of loopback" — live bind 0.0.0.0:8787, ss -ltnp 2026-10-18; off-network ingress is SSH tunnel only — user tunnel, port 22 (sshd -T → port 22, 2026-10-18), key-only auth (unverified 2026-10-18: the per-user Match block needs sudo to read), forward remote end localhost:8787).

Gotchas: sudo podman exec fails exit 125 here — root's namespace does not see rootless containers; run podman exec hermes-webui printenv HERMES_WEBUI_PASSWORD unprivileged (the ssh user owns the env files, mode 600). If rotating because a secret leaked into transcripts/logs, rotate in place via a script that fetches the live value from the running container (printenv) instead of transcribing it.

See Also