**llama.cpp** is the inference runtime for all `frank-zealot`-class endpoints. The `frank-zealot` service runs llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build, under a root systemd unit.
**One fact, one owner.** The `serverAppConfigs` repo owns every fleet fact;
`AGENTS.md` restates only what a deploy gate or the harness needs and otherwise
links. `serverAppConfigs-docs/` (Obsidian vault) carries the lore: one page per
- Base-alignment: censored (abliterated family lives on frank-dragoon)
## Resolution Rules
Official model ids are StarCraft unit names, on the wire since 2026-10-09, and since 2026-10-10 they are also the SERVICE names: `frank-archon` → `192.168.0.235:7770`, `frank-dragoon` → `192.168.0.143:7770`, `frank-zealot` → `192.168.0.143:7771`.
The default model resolves as a `defaultProvider` + `defaultModel` pair (provider must serve the model); a subagent override uses a bare model `id`, which resolves only when unique across providers. Two ports on one host need two providers — the file cannot express "same host, different port".
When renaming a service, retiring an endpoint, or moving a model: update the provider block, model `id`, `contextWindow`, `maxTokens`, `input` in `models.yml`, then fix every `modelRoles` entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As of 2026-10-06 **all three endpoints accept images** (`input: [text, image]`) — `frank-dragoon` returned to vision with the flash-server v2 swap, while `frank-archon` takes them natively and `frank-zealot` through its BF16 mmproj.
Reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3 parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` — `models.yml` sets `reasoningContentField` per endpoint to match.
There is **no audio** input anywhere in the fleet: that capability left with the FLM NPU endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's audio encoder, and left again with the Gemma4 retirement on 2026-10-01.
## Boot Persistence & Self-Healing (MANDATORY)
Every deployed, working Podman container MUST have a self-healing, boot-persistent Quadlet unit. A container that runs only from `compose.yaml` or a manual `podman run` dies silently at reboot — rootless Podman does not auto-start bare containers, and `restart: unless-stopped` does not survive the explicit stop that shutdown performs. Never report such a service as "up".
Unit requirements:
- **Boot persistence** — `[Install] WantedBy=default.target`, installed to `~/.config/containers/systemd/<name>.container`, enabled in the user manager, with linger on for the deploy user (`loginctl show-user dank -p Linger` → `Linger=yes`).
- **Self-healing** — `Restart=always` (or `on-failure`) with a short `RestartSec`; quadlet health check where the service has one.
- **Repo provenance** — the `.container` file lives in this repo (`<host>/<svc>/quadlet/`), is installed by the deployer, and is covered by the quadlet test seam. The host-side unit is generated state; the repo file is the source of truth.
Before declaring a service deployed: `systemctl --user is-enabled <name>` → `enabled`, `is-active` → `active`, and the container answers its health check.
A running container with no unit is a gap: write the unit, test it, deploy it.
This rule exists because reboots found gaps twice (2026-08-26: Caddy as a bare compose container never returned; 2026-09-11: five observability services had no units). The observability services now have units under `dankclaw/observability/quadlet/` and their compose definitions were removed — one owner per container name. The stack is **configured and delivering as of 2026-09-17**: a Grafana datasource and the `llm-fleet` dashboard are provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`, loopback :9093) → host-side Signal relay (`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN IP, the one bind a container can reach as `host.containers.internal`; iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
Prometheus targets read down for structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is a blackbox exporter. See `serverAppConfigs-docs/Infrastructure/Observability.md`.
## Benchmarking Policy (MANDATORY)
**STOP the production service BEFORE benchmarking** — these servers serve real users, and a bench beside a live service starves the GPU and corrupts results. Use the runtime's own harness: `llama-bench` for the llama.cpp endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion``timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-dragoon and frank-zealot were measured. Rolling your own bench requires asking first.
Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself.
A gufo-era law: **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
Measured 2026-09-28 on frank-dragoon at depth 32768: its corpus reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across draft widths, i.e. the same highly-predictable text every run, and 93 % of every proposal accepted. It reported 29–37 t/s decode where real traffic gives ~15 t/s, and it made `--draft-tokens 16` look 6 % faster than 7 when real text made it 14 % *slower*. Use it for shape, not for A/B. For an A/B, replay **real* text through the live server and read the `Server-Timing` header — but note the unit rides in the **name** half (`ttft;dur=402.485`), so splitting on `=` alone keys the dict `ttft;dur` and every lookup silently returns nothing. Generate **at least 128 tokens** for the measurement: at 8 tokens, `inter_token` is first-token noise and the run-to-run spread swamps every real difference.
This covers **any GPU experiment, not just benchmarks** — including a bare 2-rank RCCL/`torch.distributed` probe. On 2026-09-16 one such probe was run against the live service "because it only needs 200 MB of VRAM"; the host went hard down ~20 minutes later (no ARP reply, 100% ICMP loss from two subnets, no recovery — a power cycle was required). A hung AQL dispatch wedging the GPU driver is the plausible mechanism, and this host has precedent for going unreachable until a power cycle. If an experiment needs GPUs, stop and mask first, or use a host that serves nobody. Telling an off machine from a wedged one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a merely wedged kernel usually still answers ARP.
1. Stop and mask (self-healing would restart it):
- Dankclaw `frank-dragoon` (root unit, halogen podman on the iGPU):
NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in `/etc/systemd/system/` is itself a symlink into the checkout — mask refuses to replace it). Stop is enough for the bench window; the deploy restores the unit, so re-deploy after benching instead of unmasking.
3. Unmask and start; verify `curl http://192.168.0.<host>:7770/health`.
**CRITICAL:** never `systemctl disable` a system unit on these hosts — it deletes the unit file from `/etc/systemd/system/`, not just the symlink. `mask` symlinks to `/dev/null` and is safe/reversible.
## Subagent Lifecycle (MANDATORY)
A subagent that appears stuck must have an explicit recovery path defined before the subagent is considered deployed. "Stuck" includes: silent completion halt, repeated identical responses, or extended latency without token emission.
Recovery requirements:
- **Explicit timeout** — a hard timeout (≤30s) on the subagent's response channel, with a defined action (retry, fallback, or escalate).
- **State checkpoint** — the subagent maintains checkpoints at decision boundaries, allowing resumption from a known good state.
- **Health signal** — the subagent publishes a heartbeat (≤5s interval) that triggers the recovery path when missed.
A subagent without these three properties is a gap: write the timeout, add the checkpoint, wire the heartbeat, then deploy.
This rule exists because subagent hangs were observed in early deployments (2026-09-xx: silent completion halt; 2026-09-xx: repeated identical responses). The deployment script must verify the heartbeat channel is live before reporting "deployed".
Before declaring a subagent deployed: verify the heartbeat endpoint returns within the timeout window, and confirm the recovery path is wired (retry/fallback/escalation logic exists in the codepath).
A subagent without a heartbeat or recovery path is a gap: wire the heartbeat, add the recovery path, then deploy.`,
- The old "Benchmarking Policy (MANDATORY)" wording demanded a bench harness per
payload; superseded 2026-10-18 by the swap law above.
- A "Subagent Lifecycle (MANDATORY)" rule (heartbeat ≤5 s, checkpoints, hard
timeout) was recorded here 2026-09 and **never implemented anywhere**; it
described nothing that exists and is deleted 2026-10-18. Do not re-add it
without a real implementation.
-The fleet model inventory that used to live in this memory file drifted out of
date within hours (it still claimed Bonsai-2-**27B** at `--ctx-size 401408`
and a dragoon 524,288-position pool). Memory files do not get deploys — the
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Frank