serverAppConfigs/AGENTS.md

# AGENTS.md — serverAppConfigs Agent Rules

## Overview

This repository is the single source of truth for the two homelab servers and
the FrankBook-Pro Mac mini. It tracks the Podman container stacks and the LLM
inference fleet. One repo, one checkout per host, one deploy script. All
services run from `/home/dank/serverAppConfigs` on each host (the 2026-08-22
merge absorbed the former `vllmScripts` repo, history preserved).

- **Dankserver** (192.168.0.235) — media/library stack (RomM, Copyparty,
  Calibre-Web, Jellyfin, Plex) plus the `frank-archon` LLM service.
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
  via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
  Alloy, Pi-hole) plus TWO LLM services: `frank-dragoon` (flash-server
  container on the gfx1151 iGPU) and `frank-zealot` (llama.cpp CUDA on the
  RTX 3090). Caddy on
  Dankclaw reverse-proxies both hosts.
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
  (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
  its podman machine; all egress rides the Gluetun WireGuard netns. Deployed
  via the `frankbook-pro` handler.

## Fleet topology (ACTIVE)

| Target | Service | Runtime | Payload | Port |
|--------|---------|---------|---------|------|
| Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 |
| Dankclaw | `frank-dragoon` | halogen-flash-server 0.16.2, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 10777, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj | 7771 |

Payload identity, shape, eras, laws and rollback live in ONE page each:
[[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/<service>.env`
is the authority for every number; a page that disagrees with it is the bug.
A retired payload is a row in that page's **Eras** table — there is no page
per attempt and no separate swap-history file.

## Repository Layout

- `dankserver/` — compose.yaml + Caddyfile + .env.example for the media stack
- `dankclaw/` — compose.yaml + Caddyfile + .env.example for the AI stack
- `shared/` — homepage, Prometheus, Loki, Alloy, SearXNG configs
- `scripts/` — `deploy.sh` / `deploy-plan.py` / `deploy-remote.sh`, LLM serve
  scripts, systemd units, env tunables, bench harnesses
- `old/` — retired LLM scripts (reference only)
- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format).
  The managed skill library is distilled there one page per skill-cluster:
  `LLM/index.md` → *Cross-Cutting References*, plus `Operations/index.md`,
  `Agent/index.md`, and the Infrastructure/Dankclaw/FrankBook-Pro service
  pages (`serverAppConfigs-docs/log.md` carries the wave plan and results)
- `tests/` — bats suite + Playwright e2e

## Git Workflow — trunk-based (fleet law, aligned 2026-10-18)

All edits flow through git. Never edit files on a remote machine directly.

`main` is the only long-lived branch: no `develop`, no release branches, no
back-merges. Frank is the only dev; this repo and `dank/command-center` follow
the same regime.

1. **Branch fresh** — `git fetch`, then branch off the current `origin/main`
   (`<type>/<slug>`, type ∈ feat|fix|docs|chore). If main moves under you,
   merge it into your branch — never rebase.
2. **Source** the live config from the target machine
3. **Commit** the fetched version here (redact secrets into `.env`)
4. **Edit** locally
5. **Test** — `./tests/run.sh` must pass
6. **Push the branch and open a PR against `main`** — origin is Gitea on
   Dankclaw. Merge with `--no-ff`; **never merge a PR without an explicit,
   current OK from Frank.** **The moment the PR is open, open it in Frank's
   browser** (`xdg-open <PR-URL>` — a Firefox tab on the workstation); never
   make him hunt for it in Gitea. Releases: tag `vX.Y.Z` straight on `main`.
7. **Pull** on the target (`deploy.sh` does this)
8. **Deploy** — `scripts/deploy.sh` plans and dispatches only changed services

**Attribution footer (issues and PRs)** — every issue or PR body you open MUST end with:

```
---
cwd: <working directory> · session: <omp session id>
```

Session id from `/status`; `cwd` is that session's own working directory — not the build checkout, when they differ (note the build worktree in the body instead). Machine-made artifacts stay attributable to the box and session that spawned them.

## Deployment

```
./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankserver
./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-archon
./scripts/deploy.sh dankclaw --dry-run
```

The planner maps changed paths to services; the remote runner restarts only
the selected service and records its revision outside the checkout.
Documentation-only changes are a no-op. LLM services are selected only by
their own files — every endpoint (`frank-dragoon`/`frank-zealot`/`frank-archon`)
by its unit/env/serve trio in `scripts/` — so
unrelated
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
runs on a host without passwordless sudo; the value never enters the repo.

**`deploy.sh <target>` with no `--service` is a fleet-wide DRAIN, not a sync.**
It also walks every per-service revision marker under the deploy-state dir and
restarts each service whose marker is behind HEAD *and* whose paths changed
since that marker — so a backlog accumulated by service-scoped deploys fires
all at once, and one failing health gate aborts the run with every later
service left pending. Name the service unless a drain is what you want
(`./scripts/deploy.sh dankserver --service frank-archon`).

### Deployment completeness (MANDATORY)

A deploy is **not complete** when `deploy.sh` exits green. It requires:

1. `./tests/run.sh` — all seams green, and
2. the Playwright e2e suite (`tests/e2e/run.sh`) — runs **automatically** as
   the `[3/3]` gate at the end of `deploy.sh` against the LIVE fleet; a
   behavior-affecting deploy that fails it exits `DEPLOY INCOMPLETE`. Bypass
   only with `--skip-e2e` in an emergency. The gate runs the conditional
   `frankbook` Playwright project (the `*arr`/qBittorrent walk,
   `tests/e2e/specs/19-frankbook.spec.ts`) ONLY when a frankbook blast-radius
   path changed since origin/main — that spec, `frankbook-pro/**`,
   `shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the
   arr vhosts — so a sleeping FrankBook-Pro never fails an unrelated deploy.
   The default project set excludes it everywhere else (`E2E_INCLUDE_FRANKBOOK=1`
   opts in; see `serverAppConfigs-docs/Architecture/E2E Suite.md`).

```mermaid
graph LR
    A[Edit locally] --> B[Tests pass] --> C[Commit + push]
    C --> D[deploy.sh plan] --> E[Target pulls ff-only]
    E --> F[Dispatch selected service] --> G[Per-service health]
    G --> H[Playwright e2e gate] --> I{green?}
    I -->|yes| J[Deploy complete]
    I -->|no| K[DEPLOY INCOMPLETE]
```

## Boot Persistence & Self-Healing (MANDATORY)

Every deployed, working Podman container MUST have a self-healing,
boot-persistent Quadlet unit. A container that runs only from `compose.yaml`
or a manual `podman run` dies silently at reboot — rootless Podman does not
auto-start bare containers, and `restart: unless-stopped` does not survive the
explicit stop that shutdown performs. Never report such a service as "up".

Unit requirements:

- **Boot persistence** — `[Install] WantedBy=default.target`, installed to
  `~/.config/containers/systemd/<name>.container`, enabled in the user
  manager, with linger on for the deploy user (`loginctl show-user dank -p
  Linger` → `Linger=yes`).
- **Self-healing** — `Restart=always` (or `on-failure`) with a short
  `RestartSec`; quadlet health check where the service has one.
- **Repo provenance** — the `.container` file lives in this repo
  (`<host>/<svc>/quadlet/`), is installed by the deployer, and is covered by
  the quadlet test seam. The host-side unit is generated state; the repo file
  is the source of truth.

Before declaring a service deployed: `systemctl --user is-enabled <name>` →
`enabled`, `is-active` → `active`, and the container answers its health check.
A running container with no unit is a gap: write the unit, test it, deploy it.

This rule exists because reboots found gaps twice (2026-08-26: Caddy as a
bare compose container never returned; 2026-09-11: five observability
services had no units). The observability services now have units under
`dankclaw/observability/quadlet/` and their compose definitions were removed —
one owner per container name. The stack is **configured and delivering as of
2026-09-17**: a Grafana datasource and the `llm-fleet` dashboard are
provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/
llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are
delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/
alertmanager.container`, loopback :9093) → host-side Signal relay
(`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
IP, the one bind a container can reach as `host.containers.internal`;
iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
Prometheus targets read down for structural reasons (HTML-scraped "probes",
Gitea metrics off); the remedy is a blackbox exporter. `frank-zealot` was down for
a different reason until 2026-10-18: llama.cpp will not serve `/metrics` at all
without `--metrics`. See
`serverAppConfigs-docs/Infrastructure/Observability.md`.

## Testing

Tests are mandatory before commit (`bats` required on the dev machine):

```
./tests/run.sh
```

Seams: compose validity, env consistency (no leaked secrets), Caddyfile
validity, quadlet units, docs completeness **including vault link integrity**,
script syntax (shell + python), LLM unit integrity, LLM secret hygiene,
deployment flow, service-scoped planning, homepage config/pages/deploy/checker,
and `e2e_scaffold` — which fails if a Playwright spec file's number prefix is in
no project, because such a spec silently never runs.
See `serverAppConfigs-docs/Architecture/Testing Policy.md`.

## Secret Rules

- Never commit real secrets. The live Gitea DB password and sudo password are
  test-enforced leak checks.
- `.env` files are git-ignored; commit only `.env.example` with placeholders.
  Exception: `scripts/*.env` LLM tunables (paths and numbers, never
  credentials).
- Do not document passwords anywhere in the repo. SSH keys are the auth
  mechanism; `FRANK_SUDO_PW` covers headless sudo and never enters the repo.
- **A token passed as `curl -H "Authorization: Bearer …"` is world-readable.**
  It lands in `/proc/<pid>/cmdline`, so any `ps`/`pgrep -a` on the host prints
  it — that leaked an HF token into a session transcript on 2026-09-16, while
  merely checking whether a download was still running. Pass secrets out of
  band (`curl -K <mode-600 file>` carrying the header, or `HF_TOKEN` in the
  environment for `hf`/`hf_transfer`) and never `pgrep -a` a fetch command that
  carries an auth header.

## External Exposure Policy (MANDATORY)

Every Caddy-proxied service is either behind the auth chain (Anubis :8923 →
Authentik embedded outpost) or explicitly internal-only — no public DNS name,
no vhost. New services default to internal-only until guarded.

When guarding a new vhost — provider first, then route:

1. Create the outpost proxy provider FIRST (`external_host` = exact vhost URL
   with trailing slash; `internal_host = http://caddy:8017`). If the vhost
   flips first, authenticated traffic hits the outpost error page.
2. Flip the vhost to `reverse_proxy anubis:8923` with
   `header_up X-Real-Ip {remote_host}`, and add the post-auth
   `@<svc> host <vhost>` handler on the `http://:8017` listener routing to
   the real upstream.
3. Anubis guard set, Caddy TLS edge, and outpost provider list change
   together in one deploy.
4. Service-to-service traffic targets internal addresses (`loki:3100`), never
   the public hostname — non-browser clients cannot solve challenges.

Exempt categories (document the reason in the vhost comment): the IdP itself
(`auth.`), git smart-HTTP endpoints, services with their own complete auth.

## Deletion & Trash Policy (MANDATORY)

Never `rm`. Move deletions into trash so they stay recoverable:

1. In-repo files → `.AGENTTRASH/` beside the item's directory.
2. Elsewhere → dated staging dir `~/.AGENTTRASH/YYYYMMDD-topic`.
3. List staged items and sizes; wait for explicit user approval to purge.

`rm`, `rm -rf`, and `git clean -f` are forbidden without explicit permission.
`git * --no-verify` is forbidden. The trash is a temporary hold, not an
archive.

## Model Swaps (fleet law, 2026-10-18)

**Swap the model. Do not instrument it.** A payload change is: edit the trio
(env / serve script / unit) → `scripts/deploy.sh <host> --service <svc>` →
`/health` answers → one chat turn, one tool call, and one vision probe on an
endpoint that serves images. That is the entire procedure.

- **No benchmarking unless Frank orders it.** No pre-swap baseline battery,
  no A/B arms, no decode/prefill ladders, no acceptance tables, no numbers
  written into any page afterwards. Data exists only when he asks for it,
  and then only in the file he names.
- **It doesn't work → roll back.** Revert the commit, redeploy, done. No
  post-mortem page, no investigation log, no trial-drop-in forensic tour. If
  the fault is not obvious, one line in the endpoint page's *Laws* section is
  the whole record.
- What still gates a change: `./tests/run.sh` before commit, the deployer's
  per-service health gate, and the Playwright e2e gate. Those are the fleet's
  evidence. A bench is not evidence of correctness and buys nothing a swap
  needs.

### When a bench IS ordered

**STOP the production service first** — these servers serve real users, and a
bench beside a live service starves the GPU and corrupts the result. That
covers **any GPU experiment, not just benchmarks**, including a bare
2-rank RCCL/`torch.distributed` probe: on 2026-09-16 one was run against a
live service "because it only needs 200 MB of VRAM" and the host went hard
down ~20 minutes later (no ARP reply, 100 % ICMP loss, power cycle required;
a hung AQL dispatch wedging the driver is the plausible mechanism). Telling
an off machine from a wedged one: `ip neigh show <ip>` from a peer — FAILED
means off or NIC dead; a merely wedged kernel usually still answers ARP.

1. Stop and mask (self-healing would restart it):
   `sudo systemctl stop frank-dragoon frank-zealot` on Dankclaw,
   `sudo systemctl stop frank-archon` on Dankserver, then `mask` each.
   `mask` FAILS on a relinked unit (the file in `/etc/systemd/system/` is
   itself a symlink into the checkout); stop alone is enough for the window,
   and the deploy restores the unit — re-deploy after benching rather than
   unmasking.
2. Bench against the service itself. No scratch ports on an LLM host — one
   process owns the card. Use the runtime's own harness (`llama-bench`; a
   live llama.cpp server also reports honest `timings` per request, including
   `draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork build
   10777 that object only reaches you on **`/v1/chat/completions`**: a greedy
   `ignore_eos` run on the raw `/completion` path there answers **500 "The
   model produced output that does not match the expected Content-only
   format"** (measured 2026-10-18). halogen has no `/completion` path at all,
   so its rates are wall-clock around `/v1/chat/completions` plus the boot
   ledger. Generate ≥128 tokens: at 8, `inter_token` is first-token noise and
   run-to-run spread swamps every real difference.
3. Unmask, start, verify `curl http://192.168.0.<host>:7770/health`.
   Retire every bench drop-in the same session
   (`systemctl show <svc> -p DropInPaths` must come back empty) and write the
   numbers into the file Frank named — `/tmp` and the journal are not records.

Instrument traps that survive from the era when this was routine:
`gufo bench` cannot rank speculative decode and **inverts the ranking** (its
corpus reports `acceptance=0.933 ± 0.000` with an identical trace sha256
across draft widths — the same predictable text every run); llama.cpp's
`Server-Timing` carries the unit in the **name** half (`ttft;dur=402.485`),
so splitting on `=` keys the dict `ttft;dur` and every lookup silently
returns nothing; halogen 0.16.2 answers `/completion`, `/ping`, `/props` and
`/tokenize` with 404 and sends no `Server-Timing`; and the first request
after a boot is a warmup artifact (measured 2.6 t/s prefill then 1,258 t/s on
the next), so never report request #1.

**CRITICAL:** never `systemctl disable` a system unit on these hosts — it
deletes the unit file from `/etc/systemd/system/`, not just the symlink.
`mask` symlinks to `/dev/null` and is safe/reversible.

## Documentation Policy (fleet law, 2026-10-18)

**One fact, one owner.** A page owns exactly one thing — an endpoint, a
service, or a technique — and every other file links to it with `[[Page]]`.
Restating a shape, a digest, a port list, or a measured number in a second
file is a defect: it is the next thing to go stale.

- **One page per endpoint, forever:** `serverAppConfigs-docs/LLM/<endpoint>.md`
  — current shape, an **Eras** table (`window | payload | runtime | shape |
  replaced because | rollback`), the **Laws** the endpoint taught, the wire
  contract, and the files that own it. A payload swap edits the shape block
  and appends one Era row. It never creates a page, and the retired era's
  page goes to `.AGENTTRASH/` once its rollback path and its laws are in the
  table.
- **`log.md` records incidents and laws** — anything with a causal story
  (why the host went down, why a knob resets at reboot, why the RAM was the
  fault). A model swap is not an incident: it is an Era row and nothing else.
- **`AGENTS.md` carries operator-critical numbers only** — port, context
  ceiling, slot shape, unit/env/serve trio — plus the cross-cutting laws.
- **Index pages are locators:** one row, one clause, no numbers.
- **A measured number is written once:** in the env-file header beside the
  serve script that consumes it, and in the endpoint page. Every other page
  cites it by link.
- Technique pages (spec decode, KV math, offload budgets, PP bandwidth) stay
  technique pages: they stop carrying a per-endpoint throughput table or a live
  shape, but a campaign ledger that exists nowhere else is owned by the page that
  measured it. The defect is two pages carrying one figure, not a number living on a
  technique page — move a figure only when an endpoint page or an env header is its
  home; otherwise leave it in place and name the era it measured.

## Model Config (harness)

The harness reaches the fleet through `.omp/agent/models.yml` in the
operator's home — `/home/blank/.omp/agent/models.yml` on the dev box, which is
where the harness runs; there is no copy on the servers. Each provider is one
HOST:PORT; each model `id` is the served alias. **Official model ids are
the StarCraft unit names (2026-10-08), on the wire since 2026-10-09, and
since 2026-10-10 they are also the SERVICE names:** `frank-archon` →
`192.168.0.235:7770` (`FRANK_ARCHON_SERVED_NAME`), `frank-dragoon` →
`192.168.0.143:7770` (`HALOGEN_MODEL_ID`), `frank-zealot` →
`192.168.0.143:7771` (`ALIAS`). The 2026-10-10 cutover renamed the units,
scripts, env files, containers, deploy services and runbooks to match —
the three former service names are fully retired and exist nowhere in
the repo; historical text in runbooks uses the current names. Verify a
rename on `/v1/models`, never
on a chat response: radiance echoes the **requested** model name, so a 200
proves nothing about the served id (llama.cpp and halogen ignore it).

Resolution rules: the default model resolves as a `defaultProvider` +
`defaultModel` pair (the provider must serve the model); a subagent override
uses a bare model `id`, which resolves only when unique across providers. Two
ports on one host need two providers — the file cannot express "same host,
different port".

When renaming a service, retiring an endpoint, or moving a model: update the
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
a role pointing at a provider the file does not define fails at call time. As
of 2026-10-06 **all three endpoints accept images** (`input: [text,
image]`) — `frank-dragoon` returned to vision with the flash-server v2 swap
(the v2 vision tower loads beside the checkpoint; the text-only halogen
27B window closed 2026-10-06) — while
`frank-archon` takes them natively (radiance runs the checkpoint's
27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
are the engine core; `--mm-max-patches` available),
and `frank-zealot` through
the publisher's own **Q8_0** mmproj passed as `--mmproj` — Q8_0 is the
publisher's requirement for concurrent image requests (the card states the
BF16 tower can crash `llama-server` when several image requests run at once,
and this endpoint runs several slots). `frank-zealot`'s `contextWindow` is
**200704** — the PER-SLOT window (`--ctx-size 602112 --parallel 3` since the
2026-10-19 reslice that funded a third slot AND an armed MTP drafter),
deliberately below the model's native window; llama.cpp answers a longer prompt
with a 400 naming the limit, never a truncation, so the harness must compact
before it.
Note also
that
the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
models.yml sets `reasoningContentField` per endpoint to match.
There is **no audio**
input anywhere in the fleet: that capability left with the FLM NPU
endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
audio encoder, and left again with the Gemma4 retirement on 2026-10-01
(neither the MiMo nor the Qwen3.5 tower carries audio).

Each endpoint has ONE page in the vault: shape, Eras, Laws, wire contract,
rollback. The numbers below are only what a consumer or a deploy gate needs;
the env file beside the serve script is the authority, and a page that
disagrees with it is the bug.

### frank-dragoon — halogen flash-server, Dankclaw iGPU, :7770

- Owns: `scripts/frank-dragoon.env`, `scripts/serve-frank-dragoon.sh`,
  `scripts/frank-dragoon.service`, container `frank-dragoon-halogen`, image
  digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`.
  Engine stdout (the boot ledger: weights, KV pool, clamp) goes to
  `/home/dank/logs/frank_dragoon.log`, NOT the journal.
- Shape: `HALOGEN_CTX=200704` is the **per-request ceiling** (a longer prompt
  is a 400, never a truncation); `HALOGEN_KV_SLOTS=2` over ONE
  `HALOGEN_KV_POOL_POSITIONS=401408` pool — pool positions buy concurrency,
  the per-request window never moves; `HALOGEN_SPEC_ADAPT=0` keeps the MTP
  drafter always armed (the adaptive default disarms it and halves decode);
  server sampler 1.0/0.95/20.
- Harness contract: `models.yml` `contextWindow: 155648` = ceiling − 45,056.
  The clamp is `min(45056, window − est − 4096)` and the chars/4 estimate
  under-counts, so a harness window equal to the ceiling makes every long
  request a guaranteed 400. Reasoning arrives in `reasoning_content`; the
  template rejects `reasoning_effort` outside `xhigh|medium|low`; images are
  OpenAI `image_url` parts (probe ≥256 px); `response_format` unsupported;
  `/completion`, `/ping`, `/props`, `/tokenize` are 404.
- Page: [[frank-dragoon]].

### frank-zealot — llama.cpp (PrismML fork), RTX 3090, :7771

- Owns: `scripts/frank-zealot.env`, `scripts/serve-frank-zealot.sh`,
  `scripts/frank-zealot.service`; fork build `llama-prism-b10777` is the ONLY
  loader (the payload carries the fork's custom PQ2_0 tensor type plus
  `prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a
  loader ignoring that metadata dequantizes wrong); payload
  `Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16
  tower can crash the server with concurrent image requests).
- Shape: `--ctx-size 602112 --parallel 3` = **three conversations at
  200,704 each**, `CACHE_TYPE_K=f16` + `CACHE_TYPE_V=q8_0` (asymmetric: 25,088
  B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024,
  `SPEC_TYPE=draft-mtp` (depth 2), `--flash-attn on`, `--jinja`, `--metrics`.
- Harness/gate contract: `contextWindow: 200704` is the **per-slot** window
  (`n_ctx_slot = CTX_SIZE / PARALLEL`), and llama.cpp answers a longer prompt
  with a 400 rather than truncating. `--metrics` is mandatory or Prometheus
  reads 501 = DOWN beside a healthy service. Presence penalty has no GGUF
  metadata key, so it must reach argv. This endpoint is the fleet's
  compaction hub.
- Page: [[frank-zealot]].

### frank-archon — radiance, two R9700s TP=2, Dankserver :7770

- Owns: `scripts/frank-archon.env`, `scripts/serve-frank-archon.sh`,
  `scripts/frank-archon.service`, `scripts/ttm-preflight.sh`, template
  `scripts/qwen38-froggeric-v22.5.jinja`; image digest-pinned
  `stilldeadcode/radiance@sha256:4c3f4845…`; container name = unit name.
- Shape: TP=2 over `wht6`, `--max-num-seqs 3`, `--max-model-len 200000`,
  `expert_tiered` placement, 44,928 MiB pinned host pool per rank, VRAM pair
  `--vram-weights-mib 25600 --vram-kv-mib 5120` (ONE budget: weights + kv ≤
  30,969 MiB/card), `fp8` KV, MTP depth 2, MNBT 2048, prefix cache 8 GiB host
  + 64 GiB disk, n-gram table pinned to RAM.
- Harness/gate contract: `models.yml` `contextWindow` MUST quote 200,000
  (over-length is refused, never truncated); the ~147 GiB pinned-RAM ledger
  is why this is a root unit, and `ttm-preflight.sh` raises
  `ttm.pages_limit` itself because the kernel resets that knob to half of RAM
  on every boot; `/health` binds only after the full load, so a bare health
  gate is honest.
- Page: [[frank-archon]].

## Key Lessons Learned
Durable operational lessons from the fleet's history. Full detail in the
linked runbooks.

**PCIe / multi-GPU**

- **RCCL collectives need PCIe atomics on every participating GPU, and the failure never names
  atomics.** The kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch them on
  a device without atomics and the error surfaces as `hipErrorIllegalState` from `hipLaunchKernel`,
  not as anything that names atomics. The value is consumed at driver PROBE time, so writing the
  AtomicOp Requester bit with `setpci` at runtime changes nothing. Check the CHAIN: `lspci -vv -s
  <gpu>` and every upstream bridge must show `AtomicOpsCap: Routing+` (a `Routing-` bridge is a
  hardware wall), and the endpoint's Device Control 2 (`setpci -s <gpu> CAP_EXP+28.w`) must have bit
  6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the
  chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves
  nothing: NCCL short-circuits at `world_size=1`. [[PCIe Link and Atomics]]
- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took a board
  whose CPU lanes bifurcate to x8/x8 Gen5: both cards on root ports, every bridge `Routing+`, BOTH
  endpoints `ReqEn+`. Bandwidth was never the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB
  per token-step, under 0.4 ms at Gen5 x8); the collective choice is — TheRock's RCCL builds lack
  gfx1201 device code (`invalid device function`), which is why vLLM runs its R4D one-shot allreduce
  here at all. Evidence matrix and the three-arm verdict: [[Dual-R9700-Cooperation]].
- **When a GPU collective dies, read the runtime, not the framework.** The message that identified
  the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` →
  `AQL dispatch failed!`), never from vLLM or RCCL — the env var costs one line and prints the
  failing HIP call with its return code. [[PCIe Link and Atomics]]
- **Layer split is the no-collective path, on one vendor's cards and across two — and
  `--override-tensor` is what cannot ride with it.** Layers go in order and only the boundary
  activation crosses PCIe once per forward pass, so it works wherever a collective cannot be
  dispatched (on an RCCL-less fork it beats tensor mode, which falls back to a butterfly allreduce),
  and it is the cross-vendor answer too (a CUDA + Vulkan split held a real two-device graph). But
  `--override-tensor <group>=<device>` sets `has_tensor_overrides()`, a REQUIRED term of the
  scheduler's `pipeline_parallel` flag, and that flag sets `n_copies`: with `n_copies == 1` a
  pre-allocated weight on a non-primary device has no backend and the graph aborts (reported with
  the leaf's op name `(NONE)`). `=CPU` is the only override target that works — the CPU backend is
  last in the list — and pure Vulkan changes nothing: the gate is the override, not the driver mix.
  Pin the spec-decode drafter to one device; on a split its home is the card holding the target's
  last layers. [[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]]
- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail.**
  Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), pageable within 1 % of pinned —
  the copies are LINK-limited, so the usual "pageable costs 2×" rule does not apply; the x4 lives in
  the adapter or cable (card and port are both wider). For any engine that STREAMS weights across
  it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model needing ~1.70 GiB of active experts per
  token lands at 3.7 tok/s against 20.46 tok/s with the experts COMPUTED in host RAM
  (`-ot exps=CPU`, where the link carries activations only). Streaming loses; RAM does not care.
  [[x4-pair-experiment]]


**ROCm / GPU**

- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a trap.** With a missing
  `DST`, cp creates `DST` as a copy of `SRC` and drops one directory level. rocBLAS and hipBLASLt
  look for their device code at a FIXED depth (`../lib/rocblas/library`,
  `../lib/hipblaslt/library/<arch>`), so the flattened copy fails at REQUEST time, not load time:
  the engine reads the weights, answers `/health` and dies on the first completion with
  `Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch : gfx1201`. Declare
  the destination, then copy the contents.
- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the ~50 GiB PLE table into
  anonymous memory → silent cgroup kill, with dmesg as the only evidence. mmap pages ARE page cache,
  and pinning the PLE tensor with `-ot` cost 23 → 13.4 t/s. [[Llama Cpp Serving Sizing]]
- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve scripts export env files
  with `set -a`, so a tunable named `TEMP` leaks into HIP (`TEMP=1.0` sent the JIT cache to path
  `1.0` and segfaulted every load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a
  tunable these; check with
  `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"`.
- **`GGML_CUDA_FA_ALL_QUANTS` is dead; do not build for it.** Verified on this host 2026-10-19:
  master's `ggml/cmake/common.cmake` warns it is deprecated in favour of `GGML_CUDA_FA_QUANTS`, the
  successor's DEFAULT pair list (`q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16`) already compiles the
  quantized-KV kernels with the old flag OFF, and neither llama tree on this host references the old
  macro anywhere under `ggml/src/ggml-cuda/`. The old "~12× prefill collapse" belongs to the older
  source line. Prove a build's KV ability from its `CMakeCache.txt`. [[frank-zealot]]
- **Flash-Next MTP heads do not load on any current loader** — they need full per-layer
  `blk.<il>.hc_attn_norm.weight` and ship only `blk.48` (verified 3×). Do not re-attempt without new
  loader support.
- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on qwen4exp needs
  `scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on the agentionai FAST layer set; the
  kingjones STRIX set works unpatched. [[Llama Cpp Serving Sizing]]
- **The GPU budget on Dankclaw is the GTT aperture, not the rocm-smi "dedicated" figure.** The
  Radeon 8060S (gfx1151) reports 116,732 MiB (`llama-server --list-devices`; `gtt_total` in
  `/sys/class/drm/card1/device/`), and the older "64 GiB, SVM-backed" lesson was already stale.
  `free`/`dmidecode` show the CPU-side carve-out — intentional, do not change. [[Dankclaw System Specs]]

**Load / crash diagnosis**

- Model load is slow (minutes cold) and health returns 503 while it runs: do NOT kill the process,
  `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). On halogen the FIRST request after a (re)start
  can go silent for ~45–60 s (`engine_unresponsive`, 503s) and then self-recovers — do not restart
  over it. [[frank-dragoon]]
- Crash-loop procedure: (1) `journalctl -u <svc> -n 100` AND the kernel line via `journalctl -k` (a
  segfault at address 8 in `libamdhip64` is a NULL unref in HIP); (2) run the serve script, then the
  raw binary — a different outcome isolates the env file; (3) `gdb -batch -ex run -ex "bt 25"
  --args bash scripts/serve-<svc>.sh`, because llama-server buffers stdout and loses the log tail on
  SIGSEGV: trust gdb, not the log; (4) `amdgpu: SVM mapping failed` spam is a symptom, not a root
  cause. [[LLM Performance Diagnosis]]
- Journal signatures: a client abort logs `W srv stop: cancel task, id_task = N` while a natural stop
  logs only `slot release` with `truncated = 0`. The server never preempts a running generation;
  excess tasks queue when slots are busy.

**Host memory / swap**

- **Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff` of the old one**, or
  live pages must land in RAM that `MemAvailable` overstates. Dankclaw's 64 GiB `/swap.img`
  (fstab-driven) protects the co-tenants, not an endpoint with pinned weights — paging the KV pool
  is worse than the OOM it prevents. Host swap state is invisible to `deploy.sh` (no repo file):
  `swapon --show` before reasoning about headroom. [[Swap Configuration]]
- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults to half of RAM, so
  a manual raise vanished with a power failure, frank-archon's pinned-ledger preflight refused the
  start, and `Restart=always` re-ran that refusal 6,919 times in ~20 h — **a unit in
  `activating (auto-restart)` that exits 1 in 50 ms with no container and no engine log is a
  PREFLIGHT refusal, not an engine fault.** Repo law now: `scripts/ttm-preflight.sh` raises the knob
  to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only after a write the
  kernel rejected. Never prescribe a manual sysfs write as the recovery procedure for a service that
  can boot before a human reaches it. [[frank-archon]]
- **Silent RAM corruption outranks every software theory.** frank-archon's long-conversation
  repetition/garbage was marginal DDR5 at 4800 MT/s on four DIMMs, fixed by dropping to 4600.
  Consumer non-ECC raises NO MCE for a silent flip, and a short in-band pattern check (60 s over
  8 GiB of 246 GiB) is not a stability proof — both instruments said "ruled out" while the RAM was
  the cause, and "4800 is the JEDEC floor" is a rating, not a margin guarantee. When the fault is
  data-shaped (NaN states, garbage tokens) and a software revert does not move it, price in the RAM
  before the runtime; a revert tested under the same marginal hardware is itself confounded.
  [[frank-archon]]

**Runtime-specific guards**

- **Rootless podman cannot RAISE rlimits, so `--ulimit nofile=1048576` dies at container CREATE**,
  not at runtime: runc reports `error setting rlimit type 7: operation not permitted` (type 7 =
  RLIMIT_NOFILE) when the requested hard limit exceeds the host process's ceiling (524288 on
  Dankclaw). Inherited-equal limits are fine, so `memlock=-1:-1` works iff the unit grants
  `LimitMEMLOCK=infinity` — systemd is privileged and the value flows through podman to runc without
  any raise. Upstream halogen's own run lines pass NO `--ulimit` at all: copy vendor run lines
  before adding flags. [[frank-dragoon]]
- **A GGUF's custom tensor type picks the runtime — read the histogram before promising a file to a
  loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 over 311 tensors of type 103
  (`GGML_TYPE_Q8_0_ROCMFPX`), so stock llama.cpp and gufo refuse it at load and the fork was not a
  preference but the only door. Counting tensor types out of the GGUF KV section takes one python
  read and settles a runtime argument in minutes. Build identity matters as much as the fork: the
  2026-08-25 ROCmFPX builds SIGSEGV in `ggml_compute_forward_set_rows` on this same file, so
  `scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA. [[GGUF Compatibility]]
- **Speculative-decode flags move as a SET.** Handing the fork `--model-draft` with no `--spec-type`
  (the naive way to build a "no MTP" bench arm) segfaulted at load, core dumped, no diagnostic. A
  baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND `--spec-type`, or it
  measures a crash instead of the absence of drafting. [[frank-zealot]]
- **An orphaned scratch server taxes the whole host.** A bench child left alive by a client-side
  exception kept ~26 GiB of weights resident on a 121 GiB box and pushed `available` to 3 GiB, so
  every later rate was suspect. Record the child pid, kill it in a `finally`, then prove `free` came
  back before believing a number — and kill by pid, never `pkill -f llama-server`.
  [[LLM Bench Windows and Measurement]]
- Retired-runtime traps that can come back: **Kairic Edge** needs
  `third_party/composable_kernel/include/ck/config.h` generated manually (untracked, so `git clean`
  removes it) and its greedy-argmax fastpath rejects non-greedy requests and reasoning budgets, so
  re-test chat before production; the **FastMTP sidecar** must be commit `4df29be4` + the HauhauCS
  patch, since stock binaries fail the draft load with a tensor-shape mismatch.

**llama.cpp serving**

- **MTP is a property of the FILE, not of the flags.** The `-mtp` export carries the head as
  `blk.<n>.nextn.*` (`qwen35.nextn_predict_layers = 1`); `--spec-type draft-mtp` sets
  `load_mtp` and the draft context runs on the MAIN model's weights (`shares_model`), so there
  is no second copy of a 12 GiB model in VRAM and no `-md` argument. The plain build has no MTP
  tensors at all, and pointing `--model-draft` at that file loads it twice. [[frank-zealot]]
- **What can be chained: ONE context-backed drafter plus any number of draftless ngram types.**
  `--spec-type a,b` chains in a fixed internal order (CLI order irrelevant) and the gate is
  `ctx_dft`: the ngram family is added unconditionally while `draft-mtp`, `draft-dflash`,
  `draft-dspark` and `draft-eagle3` all need a draft CONTEXT. So `draft-dflash,ngram-mod` is
  legal (measured 2.6x the drafter alone on prompt-echo traffic) but `draft-mtp,draft-dflash`
  dies at load — *"context type MTP requested but model doesn't contain MTP layers"*. **A LEGAL
  chain can still lose on a recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35
  hybrid measured −15 % prose / −7 % block-copy, because adding the ngram layer logs
  `recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support` and
  MTP verification depends on that rollback. Measure chains per architecture, never by analogy.
  [[frank-dragoon]]
- **A drafter's KV scales with the TARGET's context, and its window policy decides the price.**
  The draft context is built at `llama_n_ctx(ctx_tgt)` with no flag to cap it, so a drafter
  declaring NO sliding window caches the whole window on `--spec-draft-device` and can OOM the
  first draft graph capture AFTER `/health` has answered. Read `sliding_window` out of the
  drafter's GGUF header before sizing, and prefer a skewed `-ts` over a context cut.
  [[Llama Cpp Multi-Device Splits]]
- **A draft context bills its own f16 KV over the WHOLE POOL, not the slot**, plus its own
  compute buffers, so on a single-card endpoint that squeeze can cost more than the drafter
  wins. The MTP question is never "does the file carry the head" but what draft KV + buffers
  cost against what the batch knobs leave for compute. Measure the arm, never the flag.
  [[frank-zealot]]
- **A quant's scale overhead belongs in its byte rate; doubling a measured figure to "get the
  f16 number" invents 6.25 %.** q8_0 is 1.0625 B/element (32 codes + one 2-byte scale per
  32-block), f16 is exactly 2.0 ⇒ the ratio is **1.882×**, not 2×. Derive byte rates from element
  count × block layout, then check them against the boot ledger — this repo carried an invented
  2× number for weeks. Current KV math: [[frank-zealot]]
- **`--ubatch-size` has an OPTIMUM, not a plateau, and the curve is BACKEND-SPECIFIC — never
  carry it across hosts.** Its compute buffers land about TWICE as heavy on the card that holds
  the output layers and the drafter, so a split balanced at one batch is unbalanced at another:
  re-measure the balance after any batch change. The logical `--batch-size` is nearly free; a
  large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit, which no totals
  model can see. A prompt shorter than the ubatch cannot see the knob at all — that is the
  control that proves the knob, not the traffic, moved the number. The 2× R9700 split peaks at
  512; the CUDA 3090 has no interior optimum up to 2048. When a bench IS ordered, A/B it by
  restarting the SERVICE through a transient `FRANK_<SVC>_ENV_FILE` drop-in, never a scratch
  port: one card holds one server. Ladders: [[frank-medium-llamacpp-dflash2]] and
  [[frank-zealot]].
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same size** (a DeltaNet
  boundary and a full-attention boundary can differ by ~1 GiB each with its own KV), so balance
  by measurement. An affine fit reproduces the deployed point and overstates the gaps in between.
  [[Llama Cpp Multi-Device Splits]]
- **`ngram-mod` prints no startup banner.** The load log names `draft-dflash` and nothing for the
  matcher, so a chain can look unengaged while it works. Identify it behaviourally (`draft_n` > 0
  only where text repeats) or by the per-task `draft acceptance = ... mean len = ...` line. It
  costs no VRAM (a 64 MiB host-RAM hash table) and its window decides which traffic it can see:
  too short and it proposes on ordinary prose fragments, costing verification without acceptance.
  [[Llama Cpp Multi-Device Splits]]
- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the per-slot number is
  the contract — not `ctx-size`.** `n_ctx_slot = CTX_SIZE / PARALLEL`, and llama.cpp refuses a
  prompt longer than the SLOT with a 400 even when the pool has room, so `models.yml`
  `contextWindow` MUST follow every reslice. Both slot directions cost or buy window, never
  pool; shorter slots decode slightly faster. Measured reslices: [[frank-zealot]].
- **Price a VRAM budget from two boots, never from a formula — and an mmproj costs ~1.5× its file
  size** (the encoder's activation and compute buffers ride with it). Subtracting two booted rows
  isolates it exactly; extrapolating weights+KV+"a bit for buffers" is what leaves 1.1 GiB free
  when you meant 2.7. Then solve `KV ≤ card − non-KV − headroom` and round to a multiple of
  `PARALLEL × 4096`. **The projector is NOT billed in llama.cpp's own fit check**: the engine
  prints its worst case and fits WITHOUT it, so price a shape from settled `nvidia-smi` after a
  real boot — the projector, and any drafter context, lands after `/health` would already have
  answered (same class as the DSpark drafter OOM). Two-boot table: [[frank-zealot]].
- **`images` and `image_url` are NOT interchangeable: only the content part reaches the model.**
  `--mmproj` loads, `usage.prompt_tokens` rises, and the model still invents an answer if the
  client used the legacy top-level `images` field on `/v1/chat/completions` — frank-zealot
  answered "Brown." for a solid-red PNG that way and "Red." through the OpenAI `image_url` part.
  The failure is silent because a hallucinated color is a plausible answer; probe with a solid
  color and expect an exact word. [[frank-zealot]]
- **`--jinja` is what `tools` needs.** A subagent died with `500 tools param requires --jinja
  flag` on the ik_llama.cpp runtime that served frank-dragoon: every tool-using agent on that
  endpoint was broken and the failure read like a model fault. Mainline llama.cpp defaults
  `--jinja` on and the serve scripts pin it; halogen also accepts `tools` and returns
  `tool_calls`. Check tool support before routing a role at an endpoint.
- **A model's own template can REJECT the harness's thinking level.** Qwen3.8's template accepts
  only `xhigh`, `medium` and `low` for `reasoning_effort` and raises on anything else — HTTP 500,
  not a clamp — so the harness's levels must be folded onto that set in `models.yml`
  (`compat.reasoningEffortMap`). A role whose level is unmapped fails on its FIRST request and
  the failure names Jinja, not the model: every subagent on `frank-zealot` died with
  `Unexpected reasoning effort high` until `high: xhigh` was added. Halogen accepts the wider
  set, so the two entries legitimately differ.
- **Thinking tokens count against `max_tokens`, so a short-budget vision probe reads as
  blindness.** A probe at `max_tokens: 64` returned `content: ""` with `finish_reason: "length"`
  and half a thought in `reasoning_content` while the image HAD reached the model (322 prompt
  tokens vs 42 for the text alone). Budget ≥512 for any thinking model you probe for vision.
- **The FIRST request after a boot is a warmup artifact**, and the 3× spread on one shape makes
  graph-capture cost read as a model fault. Benchmark the second request or later.
  [[frank-zealot]]
- **A fork build hides its own dtype laws.** Vendor builds carry their own compile options, so
  prove a build's KV behaviour from the KV buffer line in the boot ledger and its
  `CMakeCache.txt` — never from the build flags you remember passing. [[frank-zealot]]
- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed**: without it
  the engine answers **501 Not Implemented** on `/metrics`, so the scrape fails while `/health`
  answers 200 and `LlmEndpointDown` fires beside a healthy service. halogen serves `/metrics`
  natively, so the asymmetry reads as "only the 3090 is broken". Diagnose with `lastError` from
  `/api/v1/targets`, not the alert list, and note Alertmanager has no persistence: a reboot
  re-stamps `startsAt` on an alert that has fired for weeks. [[Observability]] [[frank-zealot]]
- **A leftover bench drop-in silently hijacks the next deploy, and no health gate sees it.** The
  sanctioned bench-window method (a transient `Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env`
  drop-in sourcing the repo env) outlives the bench unless retired: a leftover
  `frank-zealot.service.d/ptq1.conf` made the cutover deploy boot a **different model at a
  different shape** while `/health` answered 200, `/v1/models` echoed the alias (a flag, not a
  fact) and the Playwright gate passed 33/33. After deploying any recently-benched LLM service:
  `systemctl show <svc> -p DropInPaths` (must be empty), then
  `tr '\0' ' ' </proc/$(pidof llama-server)/cmdline`, then the boot ledger — and retire drop-ins
  to `~/.AGENTTRASH/<date>-<topic>/` + `daemon-reload`. [[frank-zealot]]
- **A bench verdict that never reaches the repo is indistinguishable from a rogue arm.** That
  same arm served production for 24 minutes with real numbers (~48 t/s decode, 801 t/s prefill at
  141k) that lived only in `/tmp` env files and journald, so the next agent read the surviving
  drop-in as an unfinished experiment and reported a hijack. Write the measurement into the
  endpoint page and the arm script's header in the session that takes it — `/tmp` and the journal
  are not records.
- **`llama-bench` separates a LIST with `/` and SWEEP values with `,`** — for both `-dev` and
  `-ts`. `-dev CUDA0,Vulkan1` is two *single-device* configurations, never one two-device run,
  and `-ts 0.85,0.15` is two sweep values each putting everything on device 0. Write
  `-dev A/B -ts x/y`. This artifact produced a wrong fleet conclusion ("CUDA + Vulkan cannot serve
  one model") from two bogus OOMs, and the bench table's own `dev` column was printing the truth
  the whole time. Any placement-dependent number needs an independent check: sample the other
  device's memory (`nvidia-smi`) while the run is loaded.

**Ops**

- Never `pkill -f` a pattern that appears in your own command line — it kills your ssh session.
  Use `pkill -x <exact-name>`.
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in committed scripts
  (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts without it.
- **ComfyUI on Dankclaw runs with GPU access** (`/dev/kfd` + `/dev/dri` via `AddDevice`,
  `GroupAdd=keep-groups`, plus the udev rule that makes those nodes world-writable — user-unit
  spawns carry no supplementary groups) and must NOT pass `--cpu`: the gfx1151 ROCm wheels exist
  and a stale device-less container was found restarted 9,146 times. [[ComfyUI]]
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its ports off the gluetun
  service. [[SearXNG]] [[Gluetun]]
- **Dankclaw's kernel command line is a repo file.** It boots **`iommu=pt`** since 2026-10-18,
  tracked in `dankclaw/host/grub`, and moves ONLY through `scripts/set-dankclaw-bootline.sh`
  (diff, install, `update-grub`, verify `grub.cfg`, `--reboot`) — never by hand-editing
  `/etc/default/grub`. It replaced `amd_iommu=off`, and the local A/B on byte-identical traffic
  says identity mode costs nothing, so the XDNA2 NPU returned for free.
  [[dankclaw-iommu-translation-off]]
- **Rootless Quadlet containers run with `--replace --rm`**: `podman logs` covers only the CURRENT
  instance; history across recreations lives in `journalctl --user -u <name>.service`. Read
  rootless storage WITHOUT sudo — it is per-user storage, so `sudo podman logs` finds nothing.
  [[Quadlet Lifecycle]]
- **A dead rootless podman pause process breaks ALL container management on the host, and
  `podman system migrate` panics in 5.7.0.** Every `podman` command fails with
  `invalid internal status`, every start fails binding the bridges
  (`failed to bind udp listener on 10.89.x.1:53`), every `Restart=always` quadlet loops — while
  container storage stays intact. What is gone is `/run/user/<uid>/libpod/tmp/alive` (a UUID
  token) and the rootless netns; deleting the stale token is not enough. Repair: **reboot**
  (`/run/user` clears and the quadlets return on their own — check `Linger=yes` and
  `WantedBy=default.target` first). The 2026-09-16 occurrence had no OOM in `journalctl -k`:
  treat the pause process as host state that can die for its own reasons. [[Quadlet Lifecycle]]
- **A container cannot reach a `Network=host` service at the host's LAN IP**: netavark DNATs
  *published* ports only, so the host bind and the `10.89.1.1` gateway are both refused from a
  bridge container while the published port answers. Use `host.containers.internal`
  (169.254.1.2). This 502'd `hermes.franksdank.cloud` until its Caddyfile upstream changed.
  [[Container Host Reachability]]
- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as `.partNofM`
  fragments** while the smaller rungs are single files, so filtering `endswith(".gguf")` makes a
  whole quant class look absent. Fetch the parts, verify each against the Hub's LFS sha256,
  concatenate in numeric order. [[Model Swap and Rollback]]
- **A fetch script must `-f` and verify, or it reports success on an access error.**
  `curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body as the "artifact": a gated
  repo yielded three 153-byte shards with identical sha256 and a cheerful `DOWNLOAD_DONE` in 20
  seconds for 104 GiB. Use `-f`/`--fail-with-body`, compare each file's byte count to the Hub's
  declared size, and check the LFS sha256 (`scripts/download-frank-medium-gguf.sh` does all three;
  ad-hoc fetches tend not to). Gated repos 403 even with the host's HF token until that account
  accepts the gate. [[Model Swap and Rollback]]
- **A scout verdict is a lead, never a finding.** In the 2026-10-19 vault de-dup campaign six
  read-only scouts worked the same pages: one classified era pages as stale *because* they
  document retired payloads (that is an era page's content), one reported files that do not exist,
  one reported duplications that grep disproved (`457.9 t/s` and the ping-RTT table are solo-held
  by `Ethernet Bandwidth Math`), one cited a line number past the end of the file it judged, and
  one drafted an Eras table with invented windows, payloads and rollback paths. Re-verify every
  claimed duplicate, every claimed staleness and every quoted holder against the file and today's
  fleet state before editing; a scout yield buys no evidence.

## Runbook Index

One page per live endpoint; every payload swap it has ever run is a row in that
page's **Eras** table, with the rollback path in the same row. There is no
per-attempt page and no per-swap ledger anywhere else.

| Service | Page |
|---------|------|
| `frank-archon` — radiance, 2× R9700 TP=2, Dankserver | `serverAppConfigs-docs/LLM/frank-archon.md` |
| `frank-dragoon` — halogen flash-server, Dankclaw iGPU | `serverAppConfigs-docs/LLM/frank-dragoon.md` |
| `frank-zealot` — llama.cpp (PrismML fork), RTX 3090 | `serverAppConfigs-docs/LLM/frank-zealot.md` |
| `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
| Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |

Retired **endpoints** (not payloads) keep their own pages — `frank-medium`,
`frank-npu`, the pre-name Dankclaw/Dankserver eras — because no live page owns
them; `serverAppConfigs-docs/index.md` is the locator for every page in the
vault. Retired scripts live in `old/` and `.AGENTTRASH/`.

## Why

Direct edits on the remote drift from version control, and future agents
won't know what changed. Git is the single source of truth. The two workloads
share two hosts, one Caddy, and one Gitea; one repo keeps them in one
history.

## Summary

| Context | Action |
|---------|--------|
| Editing a compose/Caddyfile/serve script | **Through git** — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote. |
| Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before editing. |
| Unstaged local changes | **Commit or discard** — check `git status` before starting work; ask if unsure. |
| Deploying a committed change | **`scripts/deploy.sh <target>`** — pushes, pulls ff-only, plans, dispatches, health-checks. |
| Container without a Quadlet unit | **Give it one** — boot-persistent, self-healing `.container` via git. |
| Changing a model/payload | **Swap it** — edit the trio, deploy, `health` + one chat/tool/vision probe. No benches unless Frank orders one; if it misbehaves, revert. |
| Writing a doc | **One owner per fact** — edit the endpoint/service page; everywhere else links. Never restate a number. |
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |