Files
serverAppConfigs/serverAppConfigs-docs/Dankclaw/Hermes.md
T
Frank 327e78b950 docs(vault): destroy the dead weight — 53 pages and the change diary
The vault was 155 pages / 31,210 lines, most of it measurement ledgers for
payloads that no longer run. A throughput figure for a runtime nobody operates is
not a record: AI moves faster than the vault, so the number is wrong before
anyone reads it, and a wrong number in a runbook gets believed. Per the new
purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed.

Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest):
- the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM,
  the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch
  campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two
  "Running Qwen3.6 …" pages;
- the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag)
  and the multi-device/x4-pair/hybrid/offload experiment pages;
- every vLLM page — no vLLM endpoint exists on this fleet;
- `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced
  owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff,
  self-hosted app census, the TTS survey) that were obsolete by construction;
- `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the
  page that owns the thing it governs, never to a diary.

Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras
tables keep the window, payload, runtime, reason and rollback pointer; the rate
columns are blanked. The live shapes stay verbatim — those numbers are what the
deployer and `models.yml` consume.
2026-10-19 10:56:37 -07:00

333 lines
19 KiB
Markdown

---
type: reference
description: "Hermes is the self-improving AI agent from Nous Research."
tags:
- dankclaw
- hermes
timestamp: 2026-09-15
---
# Hermes
Hermes is the self-improving AI agent from Nous Research. It runs in a terminal, Telegram, Discord, Slack, WhatsApp, or Signal, and keeps a persistent learning loop: it creates skills from experience, searches past conversations, and builds a model of the user across sessions.
**Host:** Dankclaw — the gateway itself publishes **no port** (it runs in the
host network namespace; `podman inspect systemd-hermes` → `net=host`, no port
bindings, 2026-10-18). Note the **container name drift**: the unit is
`hermes.service` but the generated start line is
`podman run --name systemd-%N …` (`systemctl --user cat hermes.service`), so
`podman ps` shows the gateway as **`systemd-hermes`** — the repo unit sets no
`ContainerName=` key even though its header comment says "container name:
hermes". The user-facing web surface is [[Hermes-WebUI]] on `:8787`, reached
`hermes.franksdank.cloud` — internal-only by DNS absence (no public Porkbun A
record; the name resolves from the Pi-hole hosts file,
`dankclaw/pihole/addn-hosts:39`) — **Docs:**
https://hermes-agent.nousresearch.com/docs
**Signal is not carrying traffic today (measured 2026-10-18).** The gateway
boots and stays up, but its own log says
`Platform 'signal' is explicitly disabled by platforms.signal.enabled: false in
config.yaml, so the credentials found in the environment (SIGNAL_HTTP_URL) will
NOT start its adapter` (`journalctl --user -u hermes` 2026-10-18), and the host
daemon it would dial is not running either: `systemctl --user
is-enabled signal-cli-daemon` → `disabled`, `is-active` → `inactive`, and
nothing listens on `:8080` (`ss -ltn`). Everything below still describes the
designed wiring; the remedy for both halves is in the log line
(`platforms.signal.enabled: true`) plus enabling the daemon unit.
**Persistent state:** `/home/dank/Containers/hermes/data` on Dankclaw, bind-mounted
at `/data` in all three Hermes containers (gateway, webui, dashboard —
`hermes.container:36`, `hermes-webui.container:46`
`hermes-dashboard.container:24`). Each
container additionally gets an anonymous volume at `/opt/data`, which is
re-created empty on every container recreation — anything that lives only there
is not durable. The tracked, sanitized configuration baseline is
[`dankclaw/hermes/config.yaml.example`](../../dankclaw/hermes/config.yaml.example).
## Facts (from research, primary source)
| Item | Value |
|------|-------|
| Image | `localhost/hermes-agent:latest`, built from `NousResearch/hermes-agent` (official Dockerfile in repo); `Pull=never` in all three units (`hermes.container:28-29`, `hermes-webui.container:35-36`, `hermes-dashboard.container:19-20`), so the tag never moves at runtime. Live image: built 2026-09-24, OCI revision label `749220ef0007f8d87bd1531f1c24b0fe93816385` (`podman image inspect localhost/hermes-agent:latest` 2026-10-18) — that build is what is running. |
| Upstream latest | agent v0.21.4 / CalVer `v2026.9.21` (2026-09-21); hermes-webui v0.52.113 (2026-08-14). Pin the CalVer tag, not `:latest`. *(unverified 2026-10-18: the local image records only a git revision, and the webui version endpoint answers 401 without a session, so neither number can be re-derived from the host.)* |
| License | MIT |
| Model backends | Nous Portal, OpenRouter, OpenAI, or any custom endpoint |
| Interfaces | TUI, Telegram, Discord, Slack, WhatsApp, Signal |
| Memory | FTS5 session search, Honcho user modeling |
| Skills | self-created, self-improving during use |
## Configuration
- Point Hermes at the local model endpoint or an external provider through its provider config.
- The gateway runs `hermes gateway run --force -v` in the **host** network
namespace (`Exec=` + `Network=host`, `dankclaw/hermes/quadlet/hermes.container:27,37`) and reaches the Signal daemon at
`http://127.0.0.1:8080` (`SIGNAL_HTTP_URL`, `hermes.container:33`). That loopback hop works only because the gateway is in
the host netns — see [[Container Network Namespaces]]. The daemon itself is a
**host** user unit (`~/.config/systemd/user/signal-cli-daemon.service`), not a
container. **Today it is `disabled` + `inactive` and nothing is bound to
`:8080`** (`systemctl --user is-enabled/is-active`, `ss -ltn`, both
2026-10-18), and the gateway's own config gate (`platforms.signal.enabled:
false`) keeps the adapter off even if the daemon came back — see the dated
note at the top of this page. When the daemon does run it binds
`0.0.0.0:8080`, i.e. the Signal HTTP API is reachable from the whole LAN.
- The data volume persists skills, memory, and session history.
- Hermes has two named local providers (the `dankclaw-large` name is confirmed
live from the gateway log — `resolve_provider_client: named custom provider
'dankclaw-large' has no resolvable api_key`, `journalctl --user -u hermes`
2026-10-18; the base URLs below are the repo record, since
`/data/config.yaml` is not readable without `sudo`):
`dankclaw-large` reaches `http://192.168.0.143:7770/v1` and exposes
`frank-dragoon` — verified from the host with `curl
http://127.0.0.1:7770/v1/models` → a single served id (2026-10-18; write
`127.0.0.1`, not `localhost`, because that port is a rootless pasta publish
and is IPv4-only)
`dankserver-small` reaches `http://192.168.0.235:7770/v1` and, **as of
2026-10-18, that endpoint serves only `frank-archon`** (`GET
http://192.168.0.235:7770/v1/models` → `[{"id":"frank-archon"}]`). The name
this page used to carry for it — `frank-medium`, renamed from `frank-small` on
2026-09-15 — is retired-era: nothing on that endpoint answers to it today, so
if the live `config.yaml` still names `frank-medium` the provider is dead
weight and requests will 404 on a nonexistent model. Diff the live config
before trusting either name. Which payload sits behind each port, and the
served id it answers to, is on [[frank-dragoon]] and [[frank-archon]].
- The live `config.yaml` contains secrets and user state and is not committed;
the tracked example is the sanitized backup of its non-secret settings.
## Volumes
| Container path | Host path (live) | Backing |
|----------------|------------------|---------|
| `/data` | `/home/dank/Containers/hermes/data` | bind mount (durable; shared by all three containers) |
| `/webui` | `/home/dank/Containers/hermes/webui` | bind mount (webui only — the `bootstrap.py` venv lives here) |
| `/opt/data` | anonymous volume | image `VOLUME`; fresh empty volume per recreation |
## Gotchas
- Hermes is under active development; pin the image tag for reproducibility. The units already pin by absence of pulling (`Pull=never`) and by building `localhost/hermes-agent:latest` locally — a rebuild, not a pull, is what changes the code.
- Model choice drives quality; the local `frank-dragoon` (Dankclaw, 7770) is the intended backend.
- Do not expose the gateway without auth — put it behind [[Caddy]] with basic auth. The gateway itself publishes no port today; the surfaces that are exposed are the webui (`hermes.` vhost, exempt by construction because the app enforces its own password — see [[Hermes-WebUI]]) and the dashboard (`hermesadmin.` vhost → `192.168.0.143:9119`, app-enforced basic auth — see [[Hermes-Dashboard]]).
## Browser automation (Chrome DevTools Protocol daemon)
The agent's browser tool is an MCP client (`mcp_tool.py` in the image) that
spawns `npx -y chrome-devtools-mcp@latest --browserUrl=http://127.0.0.1:9222`
per session; the browser itself lives in a dedicated daemon container, the
digest-pinned headless-shell unit `chrome-devtools-protocol.container` (host
network, loopback-only bind on 127.0.0.1:9222 — `ss -ltnp` 2026-10-18 shows
`127.0.0.1:9222 headless-shell` and nothing on `0.0.0.0`; image
`docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5…`
`dankclaw/hermes/quadlet/chrome-devtools-protocol.container:28`). Both consumers — the Signal
gateway and the webui server — run in the host network namespace (`podman
inspect` → `net=host` for both, 2026-10-18), so they can
dial the daemon on loopback; nothing on the LAN can reach the CDP endpoint.
`GET http://127.0.0.1:9222/json/version` answers **200** today with
`Browser: Chrome/151.0.7922.109`, `Protocol-Version: 1.3` (2026-10-18).
The wiring lives in the live `/data/config.yaml` under `mcp_servers.chrome-devtools`
(command `npx`, args `-y chrome-devtools-mcp@latest --browserUrl=http://127.0.0.1:9222`) —
that file is mode `drwx------` uid 109999, so reading it needs `sudo` (a
non-privileged read is `EACCES`, checked 2026-10-18).
The npx package cache lives in the container layer, so a recreation (not just a
restart) pays one first-call npm download; subsequent calls are cached.
## Config-home drift ("cannot connect" to models that answer curl)
`podman exec <container> printenv HERMES_HOME` for each hermes container. If
the webui reads the image default (`/opt/data`) while the gateway reads
`/data`, the webui runs the image-default template config (`provider: auto` →
openrouter.ai with no key = connect failure).
The recreation trap is the same one the volumes table shows: image `VOLUME`
points get a fresh anonymous volume on every container recreation
(`podman run --replace --rm`) — edits inside such a mount are lost on
restart/redeploy. Compare volume ids before/after a restart to confirm.
Fix: pin `Environment=HERMES_HOME=/data` in the quadlet unit (shared state
root), deploy service-scoped. Verify with the app's own code, not guesses:
```python
from api import config as C; C.resolve_model_provider("<model-id>")
# -> (model_id, provider, base_url); bare ids map to the owning provider by
# scanning the providers: dict models allow-list; @provider:model forces one.
hermes_cli.providers.resolve_user_provider(name, cfg["providers"])
# -> ProviderDef with real base_url; accepts api/url/base_url keys.
```
Named providers (`providers:` dict) are a catalog + connection source;
`custom:<slug>` connections resolve only via `custom_providers[]` entries. If
neither holds the provider name, the request has no endpoint and fails to
connect. Probe the endpoint with the **exact served model id** — the retired
FLM/NPU endpoints routed only by exact registry tag and answered HTTP 200
with `{"error":"Failed to load <tag> model!"}` for unknown tags; status-code
checks miss this class entirely.
## Live config edits
Live file: `/home/dank/Containers/hermes/data/config.yaml` (dir `drwx------`
uid 109999 — `ls -ld /home/dank/Containers/hermes/data` → `drwx------ 29
109999 109999` 2026-10-18; a non-privileged read returns `EACCES`, so all
reads/edits need `sudo`). The repo file
`dankclaw/hermes/config.yaml.example` is a sanitized backup and **can lag
live — always diff the live file first; never treat the example as truth**.
1. **Backup first:** `sudo cp -a …/config.yaml …/config.yaml.bak-YYYYMMDD`
(in-dir `.AGENTTRASH/` also exists for removals *(unverified 2026-10-18:
inside the `0700` data dir, unreadable without `sudo`)*).
2. **Apply** with surgical `sudo sed -i` or full-file write; YAML comments in
the live file are tool-generated — preserve them.
3. **Sweep stale references** across the whole data dir:
`sudo grep -rln "<old-id>" /home/dank/Containers/hermes/data
--include="*.yaml" --include="*.json"`. Session files/MEMORY.md are
historical — leave them.
4. **Derived caches:** `webui/models_cache.json` is keyed by a config
fingerprint (`_source_fingerprint.config_yaml` mtime/size); after edits it
is stale. Let it regenerate lazily or trash it — never hand-edit.
5. **Restart the model-consuming units** (they parse config at startup):
`systemctl --user restart hermes-webui.service hermes.service`.
`hermes-dashboard.service` is unaffected by routing changes.
6. **Verify behaviorally**: one tiny completion per endpoint with the exact
served id — POST `/v1/chat/completions` and read the body, never just the
status code (endpoint names rotate; see the LLM knowledge base).
7. **Sync the repo backup** (`dankclaw/hermes/config.yaml.example`) to the
live shape, redacting secrets (`password_hash`, phone account stay
live-only), then `./tests/run.sh` before commit.
`provider: custom` + `base_url` entries route by base_url and send the model
id verbatim; named providers resolve via their `api:` URL. **Renaming a
served alias anywhere (serve scripts, env ALIAS) MUST propagate to this
config in the same change** or Hermes requests 404 on nonexistent models —
this exact gap caused the 2026-09-05 outage.
## Auxiliary routing against local backends
When a Hermes feature that calls an auxiliary task (e.g. title generation)
fails against a local endpoint, the usual cause is a **request shape the
local runtime does not implement**, not a routing mistake.
Diagnose with the real call path — run the actual function inside the
container with the app's own constants (the probe cannot drift from
production):
```bash
base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \
'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \
podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \
podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py'
```
Copy the file AFTER any restart (`--replace --rm` discards the container
layer). The CLI at `/opt/hermes/hermes` uses system python and dies
`No module named 'yaml'`; use `/opt/hermes/.venv/bin/python`.
```python
import sys; sys.path.insert(0, "/opt/hermes")
from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text
from agent.auxiliary_client import call_llm
resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT})
```
What actually breaks, and the fix:
- **`response_format` (json_object / json_schema) is a hard 400 on the local
backends this task was written against** — what each endpoint accepts on the
wire is on [[frank-dragoon]] / [[frank-zealot]] / [[frank-archon]]. Config
cannot override it: `call_llm` merges the task's
`extra_body` first and then `update()`s the caller's, so the hardcoded one
wins. Only temperature and max_tokens have reactive retry paths.
- **Fix: move that one task to the Responses transport**, which never
forwards `response_format`:
```yaml
auxiliary:
title_generation:
provider: dankserver-small # names track the live config; diff first
model: frank-zealot
api_mode: codex_responses # real per-task key
extra_body:
reasoning:
effort: none # mandatory, see below
```
Before copying this: which id is served on which port is endpoint state —
[[frank-zealot]] and [[frank-archon]]. The provider/model
pairing above is the shape that was landed in 2026; re-derive it from the live
config before reusing it.
- **Disable thinking for any cap-limited task.** The adapter omits reasoning
only when config says `reasoning.enabled: false`. The
`reasoning_effort: none` *shorthand* produces exactly that and leaves the
server default (thinking ON) — the 64-token title budget gets spent on the
think: empty content, `finish_reason: length`. The explicit
`extra_body.reasoning.effort: none` instead sends
`{"effort":"none","summary":"auto"}`, which the local `/v1/responses`
honors.
- Both local runtimes also answer `/v1/responses` and `/v1/messages` — no
proxy needed for the alternate transport.
Verify and land: re-run the probe with **no** `api_mode`/reasoning kwargs —
it must pass from config alone (that is the production path); keep the live
file and the repo example in step; `./tests/run.sh`, commit, push — a Hermes
config change is docs+example only, no service deploy needed.
Gotchas: `auxiliary.<task>.enabled: false` is checked BEFORE
`apply_instant_title`, so disabling a broken title task also removes the
derived title — it does not degrade gracefully the way a failure does. A
local call can also fail at the model level: a 200 with an error-text body,
or a chatty non-title answer. Always read the raw content, not the status.
## Webui password rotation ("Hermex")
Auth contract (verified live 2026-08-31, re-checked 2026-10-18): `POST /api/auth/login` with JSON
`{"password":"…"}` → 200 + signed HttpOnly `hermes_session` cookie (24 h TTL)
on match, 401 on miss. No username field. `/health` answers 200
unauthenticated (`curl http://127.0.0.1:8787/health` → 200 with
`{"status":"ok", …}` today) and `/` answers 302 (`curl -o /dev/null -w
%{http_code} http://127.0.0.1:8787/` → 302 today) — **never verify auth
those**; probe
login with a deliberately wrong password (expect 401) and the real one
(expect 200). Details on the webui surface: [[Hermes-WebUI]].
1. Run the whole probe as a heredoc'd bash script
(`ssh -o BatchMode=yes dankclaw 'bash -s'`) — the host's zsh mangles
inline `${…}` even inside single-quoted ssh args; never interpolate curl
`-w %{http_code}` inline.
2. Update BOTH env files with
`sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=<new>/"`:
- `/home/dank/serverAppConfigs/.env` — the repo-root checkout's env file
(exists, `-rw------- dank dank`, 2026-10-18) but **not** loaded by this
unit: keep it in sync only so other consumers do not drift.
- `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one, for
both `[Container]` and `[Service]`**: `systemctl --user show hermes-webui
-p EnvironmentFiles` reports exactly
`/home/dank/serverAppConfigs/dankclaw/.env` (2026-10-18;
`dankclaw/hermes/quadlet/hermes-webui.container:38,62`). That is what the
container reads at create; rotating only the root file leaves the running
container serving the old secret.
3. `systemctl --user restart hermes-webui.service` (unit recreation re-reads
env; no deploy needed if the unit file itself didn't change).
4. Verify wrong-pw → 401, right-pw → 200, from host loopback **and** a LAN
client at `http://192.168.0.143:8787` (webui publishes on the LAN
interface since commit `4443897`, 2026-08-31 "publish webui on LAN interface
instead of loopback" — live bind `0.0.0.0:8787`, `ss -ltnp` 2026-10-18;
off-network ingress is SSH tunnel only —
user `tunnel`, port **22** (`sshd -T` → `port 22`, 2026-10-18), key-only auth
- (unverified 2026-10-18: the per-user `Match` block needs `sudo` to read)*
forward remote end
localhost:8787).
Gotchas: `sudo podman exec` fails exit 125 here — root's namespace does not
see rootless containers; run `podman exec hermes-webui printenv
HERMES_WEBUI_PASSWORD` unprivileged (the ssh user owns the env files, mode
600). If rotating because a secret leaked into transcripts/logs, rotate in
place via a script that fetches the live value from the running container
(`printenv`) instead of transcribing it.
## See Also
- [[Hermes-Dashboard]] — the LAN browser management UI
- [[Hermes-WebUI]] — the Hermex server (phone-facing API, SSH-tunneled)
- [[Dankclaw Stack]]
- [[OpenWebUI]] — the chat UI that pairs with Hermes
- Knowledge base: [[SearXNG]] (Hermes as the search agent)
- [[Container Network Namespaces]] — why the gateway and webui can dial loopback services and the dashboard cannot