serverAppConfigs/AGENTS.md

# AGENTS.md — serverAppConfigs Agent Rules

## Overview

This repository is the single source of truth for the two homelab servers and
the FrankBook-Pro Mac mini. It tracks the Podman container stacks and the LLM
inference fleet. One repo, one checkout per host, one deploy script. All
services run from `/home/dank/serverAppConfigs` on each host (the 2026-08-22
merge absorbed the former `vllmScripts` repo, history preserved).

- **Dankserver** (192.168.0.235) — media/library stack (RomM, Copyparty,
  Calibre-Web, Jellyfin, Plex) plus the `frank-archon` LLM service.
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
  via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
  Alloy, Pi-hole) plus TWO LLM services: `frank-dragoon` (flash-server
  container on the gfx1151 iGPU) and `frank-zealot` (llama.cpp CUDA on the
  RTX 3090). Caddy on
  Dankclaw reverse-proxies both hosts.
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
  (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
  its podman machine; all egress rides the Gluetun WireGuard netns. Deployed
  via the `frankbook-pro` handler.

## Fleet topology (ACTIVE)

| Target | Service | Runtime | Payload | Port |
|--------|---------|---------|---------|------|
| Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 |
| Dankclaw | `frank-dragoon` | halogen-flash-server 0.16.2, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 10777, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj | 7771 |

Payload identity, shape, eras, laws and rollback live in ONE page each:
[[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/<service>.env`
is the authority for every number; a page that disagrees with it is the bug.
A retired payload is a row in that page's **Eras** table — there is no page
per attempt and no separate swap-history file.

## Repository Layout

- `dankserver/` — compose.yaml + Caddyfile + .env.example for the media stack
- `dankclaw/` — compose.yaml + Caddyfile + .env.example for the AI stack
- `shared/` — homepage, Prometheus, Loki, Alloy, SearXNG configs
- `scripts/` — `deploy.sh` / `deploy-plan.py` / `deploy-remote.sh`, LLM serve
  scripts, systemd units, env tunables, bench harnesses
- `old/` — retired LLM scripts (reference only)
- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format).
  The managed skill library is distilled there one page per skill-cluster:
  `LLM/index.md` → *Cross-Cutting References*, plus `Operations/index.md`,
  `Agent/index.md`, and the Infrastructure/Dankclaw/FrankBook-Pro service
  pages (`serverAppConfigs-docs/log.md` carries the wave plan and results)
- `tests/` — bats suite + Playwright e2e

## Git Workflow — trunk-based (fleet law, aligned 2026-10-18)

All edits flow through git. Never edit files on a remote machine directly.

`main` is the only long-lived branch: no `develop`, no release branches, no
back-merges. Frank is the only dev; this repo and `dank/command-center` follow
the same regime.

1. **Branch fresh** — `git fetch`, then branch off the current `origin/main`
   (`<type>/<slug>`, type ∈ feat|fix|docs|chore). If main moves under you,
   merge it into your branch — never rebase.
2. **Source** the live config from the target machine
3. **Commit** the fetched version here (redact secrets into `.env`)
4. **Edit** locally
5. **Test** — `./tests/run.sh` must pass
6. **Push the branch and open a PR against `main`** — origin is Gitea on
   Dankclaw. Merge with `--no-ff`; **never merge a PR without an explicit,
   current OK from Frank.** **The moment the PR is open, open it in Frank's
   browser** (`xdg-open <PR-URL>` — a Firefox tab on the workstation); never
   make him hunt for it in Gitea. Releases: tag `vX.Y.Z` straight on `main`.
7. **Pull** on the target (`deploy.sh` does this)
8. **Deploy** — `scripts/deploy.sh` plans and dispatches only changed services

**Attribution footer (issues and PRs)** — every issue or PR body you open MUST end with:

```
---
cwd: <working directory> · session: <omp session id>
```

Session id from `/status`; `cwd` is that session's own working directory — not the build checkout, when they differ (note the build worktree in the body instead). Machine-made artifacts stay attributable to the box and session that spawned them.

## Deployment

```
./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankserver
./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-archon
./scripts/deploy.sh dankclaw --dry-run
```

The planner maps changed paths to services; the remote runner restarts only
the selected service and records its revision outside the checkout.
Documentation-only changes are a no-op. LLM services are selected only by
their own files — every endpoint (`frank-dragoon`/`frank-zealot`/`frank-archon`)
by its unit/env/serve trio in `scripts/` — so
unrelated
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
runs on a host without passwordless sudo; the value never enters the repo.

**`deploy.sh <target>` with no `--service` is a fleet-wide DRAIN, not a sync.**
It also walks every per-service revision marker under the deploy-state dir and
restarts each service whose marker is behind HEAD *and* whose paths changed
since that marker — so a backlog accumulated by service-scoped deploys fires
all at once, and one failing health gate aborts the run with every later
service left pending. Name the service unless a drain is what you want
(`./scripts/deploy.sh dankserver --service frank-archon`).

### Deployment completeness (MANDATORY)

A deploy is **not complete** when `deploy.sh` exits green. It requires:

1. `./tests/run.sh` — all seams green, and
2. the Playwright e2e suite (`tests/e2e/run.sh`) — runs **automatically** as
   the `[3/3]` gate at the end of `deploy.sh` against the LIVE fleet; a
   behavior-affecting deploy that fails it exits `DEPLOY INCOMPLETE`. Bypass
   only with `--skip-e2e` in an emergency. The gate runs the conditional
   `frankbook` Playwright project (the `*arr`/qBittorrent walk,
   `tests/e2e/specs/19-frankbook.spec.ts`) ONLY when a frankbook blast-radius
   path changed since origin/main — that spec, `frankbook-pro/**`,
   `shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the
   arr vhosts — so a sleeping FrankBook-Pro never fails an unrelated deploy.
   The default project set excludes it everywhere else (`E2E_INCLUDE_FRANKBOOK=1`
   opts in; see `serverAppConfigs-docs/Architecture/E2E Suite.md`).

```mermaid
graph LR
    A[Edit locally] --> B[Tests pass] --> C[Commit + push]
    C --> D[deploy.sh plan] --> E[Target pulls ff-only]
    E --> F[Dispatch selected service] --> G[Per-service health]
    G --> H[Playwright e2e gate] --> I{green?}
    I -->|yes| J[Deploy complete]
    I -->|no| K[DEPLOY INCOMPLETE]
```

## Boot Persistence & Self-Healing (MANDATORY)

Every deployed, working Podman container MUST have a self-healing,
boot-persistent Quadlet unit. A container that runs only from `compose.yaml`
or a manual `podman run` dies silently at reboot — rootless Podman does not
auto-start bare containers, and `restart: unless-stopped` does not survive the
explicit stop that shutdown performs. Never report such a service as "up".

Unit requirements:

- **Boot persistence** — `[Install] WantedBy=default.target`, installed to
  `~/.config/containers/systemd/<name>.container`, enabled in the user
  manager, with linger on for the deploy user (`loginctl show-user dank -p
  Linger` → `Linger=yes`).
- **Self-healing** — `Restart=always` (or `on-failure`) with a short
  `RestartSec`; quadlet health check where the service has one.
- **Repo provenance** — the `.container` file lives in this repo
  (`<host>/<svc>/quadlet/`), is installed by the deployer, and is covered by
  the quadlet test seam. The host-side unit is generated state; the repo file
  is the source of truth.

Before declaring a service deployed: `systemctl --user is-enabled <name>` →
`enabled`, `is-active` → `active`, and the container answers its health check.
A running container with no unit is a gap: write the unit, test it, deploy it.

This rule exists because reboots found gaps twice (2026-08-26: Caddy as a
bare compose container never returned; 2026-09-11: five observability
services had no units). The observability services now have units under
`dankclaw/observability/quadlet/` and their compose definitions were removed —
one owner per container name. The stack is **configured and delivering as of
2026-09-17**: a Grafana datasource and the `llm-fleet` dashboard are
provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/
llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are
delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/
alertmanager.container`, loopback :9093) → host-side Signal relay
(`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
IP, the one bind a container can reach as `host.containers.internal`;
iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
Prometheus targets read down for structural reasons (HTML-scraped "probes",
Gitea metrics off); the remedy is a blackbox exporter. `frank-zealot` was down for
a different reason until 2026-10-18: llama.cpp will not serve `/metrics` at all
without `--metrics`. See
`serverAppConfigs-docs/Infrastructure/Observability.md`.

## Testing

Tests are mandatory before commit (`bats` required on the dev machine):

```
./tests/run.sh
```

Seams: compose validity, env consistency (no leaked secrets), Caddyfile
validity, quadlet units, docs completeness **including vault link integrity**,
script syntax (shell + python), LLM unit integrity, LLM secret hygiene,
deployment flow, service-scoped planning, homepage config/pages/deploy/checker,
and `e2e_scaffold` — which fails if a Playwright spec file's number prefix is in
no project, because such a spec silently never runs.
See `serverAppConfigs-docs/Architecture/Testing Policy.md`.

## Secret Rules

- Never commit real secrets. The live Gitea DB password and sudo password are
  test-enforced leak checks.
- `.env` files are git-ignored; commit only `.env.example` with placeholders.
  Exception: `scripts/*.env` LLM tunables (paths and numbers, never
  credentials).
- Do not document passwords anywhere in the repo. SSH keys are the auth
  mechanism; `FRANK_SUDO_PW` covers headless sudo and never enters the repo.
- **A token passed as `curl -H "Authorization: Bearer …"` is world-readable.**
  It lands in `/proc/<pid>/cmdline`, so any `ps`/`pgrep -a` on the host prints
  it — that leaked an HF token into a session transcript on 2026-09-16, while
  merely checking whether a download was still running. Pass secrets out of
  band (`curl -K <mode-600 file>` carrying the header, or `HF_TOKEN` in the
  environment for `hf`/`hf_transfer`) and never `pgrep -a` a fetch command that
  carries an auth header.

## External Exposure Policy (MANDATORY)

Every Caddy-proxied service is either behind the auth chain (Anubis :8923 →
Authentik embedded outpost) or explicitly internal-only — no public DNS name,
no vhost. New services default to internal-only until guarded.

When guarding a new vhost — provider first, then route:

1. Create the outpost proxy provider FIRST (`external_host` = exact vhost URL
   with trailing slash; `internal_host = http://caddy:8017`). If the vhost
   flips first, authenticated traffic hits the outpost error page.
2. Flip the vhost to `reverse_proxy anubis:8923` with
   `header_up X-Real-Ip {remote_host}`, and add the post-auth
   `@<svc> host <vhost>` handler on the `http://:8017` listener routing to
   the real upstream.
3. Anubis guard set, Caddy TLS edge, and outpost provider list change
   together in one deploy.
4. Service-to-service traffic targets internal addresses (`loki:3100`), never
   the public hostname — non-browser clients cannot solve challenges.

Exempt categories (document the reason in the vhost comment): the IdP itself
(`auth.`), git smart-HTTP endpoints, services with their own complete auth.

## Deletion & Trash Policy (MANDATORY)

Never `rm`. Move deletions into trash so they stay recoverable:

1. In-repo files → `.AGENTTRASH/` beside the item's directory.
2. Elsewhere → dated staging dir `~/.AGENTTRASH/YYYYMMDD-topic`.
3. List staged items and sizes; wait for explicit user approval to purge.

`rm`, `rm -rf`, and `git clean -f` are forbidden without explicit permission.
`git * --no-verify` is forbidden. The trash is a temporary hold, not an
archive.

## Model Swaps (fleet law, 2026-10-18)

**Swap the model. Do not instrument it.** A payload change is: edit the trio
(env / serve script / unit) → `scripts/deploy.sh <host> --service <svc>` →
`/health` answers → one chat turn, one tool call, and one vision probe on an
endpoint that serves images. That is the entire procedure.

- **No benchmarking unless Frank orders it.** No pre-swap baseline battery,
  no A/B arms, no decode/prefill ladders, no acceptance tables, no numbers
  written into any page afterwards. Data exists only when he asks for it,
  and then only in the file he names.
- **It doesn't work → roll back.** Revert the commit, redeploy, done. No
  post-mortem page, no investigation log, no trial-drop-in forensic tour. If
  the fault is not obvious, one line in the endpoint page's *Laws* section is
  the whole record.
- What still gates a change: `./tests/run.sh` before commit, the deployer's
  per-service health gate, and the Playwright e2e gate. Those are the fleet's
  evidence. A bench is not evidence of correctness and buys nothing a swap
  needs.

### When a bench IS ordered

**STOP the production service first** — these servers serve real users, and a
bench beside a live service starves the GPU and corrupts the result. That
covers **any GPU experiment, not just benchmarks**, including a bare
2-rank RCCL/`torch.distributed` probe: on 2026-09-16 one was run against a
live service "because it only needs 200 MB of VRAM" and the host went hard
down ~20 minutes later (no ARP reply, 100 % ICMP loss, power cycle required;
a hung AQL dispatch wedging the driver is the plausible mechanism). Telling
an off machine from a wedged one: `ip neigh show <ip>` from a peer — FAILED
means off or NIC dead; a merely wedged kernel usually still answers ARP.

1. Stop and mask (self-healing would restart it):
   `sudo systemctl stop frank-dragoon frank-zealot` on Dankclaw,
   `sudo systemctl stop frank-archon` on Dankserver, then `mask` each.
   `mask` FAILS on a relinked unit (the file in `/etc/systemd/system/` is
   itself a symlink into the checkout); stop alone is enough for the window,
   and the deploy restores the unit — re-deploy after benching rather than
   unmasking.
2. Bench against the service itself. No scratch ports on an LLM host — one
   process owns the card. Use the runtime's own harness (`llama-bench`; a
   live llama.cpp server also reports honest `timings` per request, including
   `draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork build
   10777 that object only reaches you on **`/v1/chat/completions`**: a greedy
   `ignore_eos` run on the raw `/completion` path there answers **500 "The
   model produced output that does not match the expected Content-only
   format"** (measured 2026-10-18). halogen has no `/completion` path at all,
   so its rates are wall-clock around `/v1/chat/completions` plus the boot
   ledger. Generate ≥128 tokens: at 8, `inter_token` is first-token noise and
   run-to-run spread swamps every real difference.
3. Unmask, start, verify `curl http://192.168.0.<host>:7770/health`.
   Retire every bench drop-in the same session
   (`systemctl show <svc> -p DropInPaths` must come back empty) and write the
   numbers into the file Frank named — `/tmp` and the journal are not records.

Instrument traps that survive from the era when this was routine:
`gufo bench` cannot rank speculative decode and **inverts the ranking** (its
corpus reports `acceptance=0.933 ± 0.000` with an identical trace sha256
across draft widths — the same predictable text every run); llama.cpp's
`Server-Timing` carries the unit in the **name** half (`ttft;dur=402.485`),
so splitting on `=` keys the dict `ttft;dur` and every lookup silently
returns nothing; halogen 0.16.2 answers `/completion`, `/ping`, `/props` and
`/tokenize` with 404 and sends no `Server-Timing`; and the first request
after a boot is a warmup artifact (measured 2.6 t/s prefill then 1,258 t/s on
the next), so never report request #1.

**CRITICAL:** never `systemctl disable` a system unit on these hosts — it
deletes the unit file from `/etc/systemd/system/`, not just the symlink.
`mask` symlinks to `/dev/null` and is safe/reversible.

## Documentation Policy (fleet law, 2026-10-18)

**One fact, one owner.** A page owns exactly one thing — an endpoint, a
service, or a technique — and every other file links to it with `[[Page]]`.
Restating a shape, a digest, a port list, or a measured number in a second
file is a defect: it is the next thing to go stale.

- **One page per endpoint, forever:** `serverAppConfigs-docs/LLM/<endpoint>.md`
  — current shape, an **Eras** table (`window | payload | runtime | shape |
  replaced because | rollback`), the **Laws** the endpoint taught, the wire
  contract, and the files that own it. A payload swap edits the shape block
  and appends one Era row. It never creates a page, and the retired era's
  page goes to `.AGENTTRASH/` once its rollback path and its laws are in the
  table.
- **`log.md` records incidents and laws** — anything with a causal story
  (why the host went down, why a knob resets at reboot, why the RAM was the
  fault). A model swap is not an incident: it is an Era row and nothing else.
- **`AGENTS.md` carries operator-critical numbers only** — port, context
  ceiling, slot shape, unit/env/serve trio — plus the cross-cutting laws.
- **Index pages are locators:** one row, one clause, no numbers.
- **A measured number is written once:** in the env-file header beside the
  serve script that consumes it, and in the endpoint page. Every other page
  cites it by link.
- Technique pages (spec decode, KV math, offload budgets, PP bandwidth) stay
  technique pages: they stop carrying a per-endpoint throughput table or a live
  shape, but a campaign ledger that exists nowhere else is owned by the page that
  measured it. The defect is two pages carrying one figure, not a number living on a
  technique page — move a figure only when an endpoint page or an env header is its
  home; otherwise leave it in place and name the era it measured.

## Model Config (harness)

The harness reaches the fleet through `.omp/agent/models.yml` in the
operator's home — `/home/blank/.omp/agent/models.yml` on the dev box, which is
where the harness runs; there is no copy on the servers. Each provider is one
HOST:PORT; each model `id` is the served alias. **Official model ids are
the StarCraft unit names (2026-10-08), on the wire since 2026-10-09, and
since 2026-10-10 they are also the SERVICE names:** `frank-archon` →
`192.168.0.235:7770` (`FRANK_ARCHON_SERVED_NAME`), `frank-dragoon` →
`192.168.0.143:7770` (`HALOGEN_MODEL_ID`), `frank-zealot` →
`192.168.0.143:7771` (`ALIAS`). The 2026-10-10 cutover renamed the units,
scripts, env files, containers, deploy services and runbooks to match —
the three former service names are fully retired and exist nowhere in
the repo; historical text in runbooks uses the current names. Verify a
rename on `/v1/models`, never
on a chat response: radiance echoes the **requested** model name, so a 200
proves nothing about the served id (llama.cpp and halogen ignore it).

Resolution rules: the default model resolves as a `defaultProvider` +
`defaultModel` pair (the provider must serve the model); a subagent override
uses a bare model `id`, which resolves only when unique across providers. Two
ports on one host need two providers — the file cannot express "same host,
different port".

When renaming a service, retiring an endpoint, or moving a model: update the
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
a role pointing at a provider the file does not define fails at call time. As
of 2026-10-06 **all three endpoints accept images** (`input: [text,
image]`) — `frank-dragoon` returned to vision with the flash-server v2 swap
(the v2 vision tower loads beside the checkpoint; the text-only halogen
27B window closed 2026-10-06) — while
`frank-archon` takes them natively (radiance runs the checkpoint's
27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
are the engine core; `--mm-max-patches` available),
and `frank-zealot` through
the publisher's own **Q8_0** mmproj passed as `--mmproj` — Q8_0 is the
publisher's requirement for concurrent image requests (the card states the
BF16 tower can crash `llama-server` when several image requests run at once,
and this endpoint runs several slots). `frank-zealot`'s `contextWindow` is
**200704** — the PER-SLOT window (`--ctx-size 602112 --parallel 3` since the
2026-10-19 reslice that funded a third slot AND an armed MTP drafter),
deliberately below the model's native window; llama.cpp answers a longer prompt
with a 400 naming the limit, never a truncation, so the harness must compact
before it.
Note also
that
the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
models.yml sets `reasoningContentField` per endpoint to match.
There is **no audio**
input anywhere in the fleet: that capability left with the FLM NPU
endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
audio encoder, and left again with the Gemma4 retirement on 2026-10-01
(neither the MiMo nor the Qwen3.5 tower carries audio).

Each endpoint has ONE page in the vault: shape, Eras, Laws, wire contract,
rollback. The numbers below are only what a consumer or a deploy gate needs;
the env file beside the serve script is the authority, and a page that
disagrees with it is the bug.

### frank-dragoon — halogen flash-server, Dankclaw iGPU, :7770

- Owns: `scripts/frank-dragoon.env`, `scripts/serve-frank-dragoon.sh`,
  `scripts/frank-dragoon.service`, container `frank-dragoon-halogen`, image
  digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`.
  Engine stdout (the boot ledger: weights, KV pool, clamp) goes to
  `/home/dank/logs/frank_dragoon.log`, NOT the journal.
- Shape: `HALOGEN_CTX=200704` is the **per-request ceiling** (a longer prompt
  is a 400, never a truncation); `HALOGEN_KV_SLOTS=2` over ONE
  `HALOGEN_KV_POOL_POSITIONS=401408` pool — pool positions buy concurrency,
  the per-request window never moves; `HALOGEN_SPEC_ADAPT=0` keeps the MTP
  drafter always armed (the adaptive default disarms it and halves decode);
  server sampler 1.0/0.95/20.
- Harness contract: `models.yml` `contextWindow: 155648` = ceiling − 45,056.
  The clamp is `min(45056, window − est − 4096)` and the chars/4 estimate
  under-counts, so a harness window equal to the ceiling makes every long
  request a guaranteed 400. Reasoning arrives in `reasoning_content`; the
  template rejects `reasoning_effort` outside `xhigh|medium|low`; images are
  OpenAI `image_url` parts (probe ≥256 px); `response_format` unsupported;
  `/completion`, `/ping`, `/props`, `/tokenize` are 404.
- Page: [[frank-dragoon]].

### frank-zealot — llama.cpp (PrismML fork), RTX 3090, :7771

- Owns: `scripts/frank-zealot.env`, `scripts/serve-frank-zealot.sh`,
  `scripts/frank-zealot.service`; fork build `llama-prism-b10777` is the ONLY
  loader (the payload carries the fork's custom PQ2_0 tensor type plus
  `prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a
  loader ignoring that metadata dequantizes wrong); payload
  `Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16
  tower can crash the server with concurrent image requests).
- Shape: `--ctx-size 602112 --parallel 3` = **three conversations at
  200,704 each**, `CACHE_TYPE_K=f16` + `CACHE_TYPE_V=q8_0` (asymmetric: 25,088
  B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024,
  `SPEC_TYPE=draft-mtp` (depth 2), `--flash-attn on`, `--jinja`, `--metrics`.
- Harness/gate contract: `contextWindow: 200704` is the **per-slot** window
  (`n_ctx_slot = CTX_SIZE / PARALLEL`), and llama.cpp answers a longer prompt
  with a 400 rather than truncating. `--metrics` is mandatory or Prometheus
  reads 501 = DOWN beside a healthy service. Presence penalty has no GGUF
  metadata key, so it must reach argv. This endpoint is the fleet's
  compaction hub.
- Page: [[frank-zealot]].

### frank-archon — radiance, two R9700s TP=2, Dankserver :7770

- Owns: `scripts/frank-archon.env`, `scripts/serve-frank-archon.sh`,
  `scripts/frank-archon.service`, `scripts/ttm-preflight.sh`, template
  `scripts/qwen38-froggeric-v22.5.jinja`; image digest-pinned
  `stilldeadcode/radiance@sha256:4c3f4845…`; container name = unit name.
- Shape: TP=2 over `wht6`, `--max-num-seqs 3`, `--max-model-len 200000`,
  `expert_tiered` placement, 44,928 MiB pinned host pool per rank, VRAM pair
  `--vram-weights-mib 25600 --vram-kv-mib 5120` (ONE budget: weights + kv ≤
  30,969 MiB/card), `fp8` KV, MTP depth 2, MNBT 2048, prefix cache 8 GiB host
  + 64 GiB disk, n-gram table pinned to RAM.
- Harness/gate contract: `models.yml` `contextWindow` MUST quote 200,000
  (over-length is refused, never truncated); the ~147 GiB pinned-RAM ledger
  is why this is a root unit, and `ttm-preflight.sh` raises
  `ttm.pages_limit` itself because the kernel resets that knob to half of RAM
  on every boot; `/health` binds only after the full load, so a bare health
  gate is honest.
- Page: [[frank-archon]].

## Key Lessons Learned
Durable operational lessons from the fleet's history. Full detail in the
linked runbooks.

**PCIe / multi-GPU**

- **RCCL collectives need PCIe atomics on every participating GPU.** RCCL's
  kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch
  them when the device has no atomics, and the failure surfaces as
  `hipErrorIllegalState` ("the operation cannot be performed in the present
  state") from `hipLaunchKernel`, not as anything that names atomics. The
  value is consumed at driver probe time, so writing the AtomicOp Requester
  bit with `setpci` at runtime changes nothing. Check the *chain*:
  `lspci -vv -s <gpu>` and every upstream bridge must show
  `AtomicOpsCap: Routing+` (a `Routing-` bridge is a hardware wall), and the
  endpoint's Device Control 2 (`setpci -s <gpu> CAP_EXP+28.w`) must have bit
  6 set. AMD desktop platforms put one card on CPU root ports (atomics work)
  and the rest behind the chipset (usually not) — that asymmetry is what
  blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL
  short-circuits at `world_size=1`.
- **That wall was physical, and it was fixed on 2026-09-20 by a board swap.**
  Both paths that put two cards on CPU root ports were: an M.2→PCIe x4 riser
  into a *CPU-attached* M.2 slot, or a board whose CPU lanes bifurcate to
  x8/x8. dankserver took the second: **ASUS ProArt B850-Creator WiFi NEO**,
  `PCIEX16_1` + `PCIEX16_2` at x8/x8 Gen5 on root ports `00:01.1` / `00:01.3`,
  every bridge now `Routing+` and **both endpoints `ReqEn+`** — the exact
  precondition the old test plan named. Bandwidth was never the constraint (a
  27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4
  and under 0.4 ms at Gen5 x8).
  **Proven on the same day.** vLLM's R4D one-shot allreduce runs TP=2 here and
  frank-medium serves on it; llama.cpp's RCCL path was built too
  (`GGML_HIP_RCCL=ON`) and its tensor mode measured — real collective, slower
  shape. Note the collective choice matters: TheRock's RCCL builds lack gfx1201
  device code (`invalid device function`), which is why R4D is the path vLLM
  uses at all. Evidence matrix and the measured three-arm verdict:
  [[Dual-R9700-Cooperation]].
- **When a GPU collective dies, read the runtime, not the framework.** The
  message that identified the cause came from `AMD_LOG_LEVEL=3`
  (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch
  failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and
  prints the failing HIP call with its return code.
- **The software path that puts both cards on ONE model without a collective:
  llama.cpp's layer split.** On the old board vLLM could not do it in any of its
  parallelisms — TP needs an allreduce, PP needs the same p2p primitive, and both
  died on the card that could not dispatch an RCCL kernel. `--split-mode
  layer` in llama.cpp needs no
  collective: layers are distributed in order and only the boundary activation
  crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill /
  18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly
  fallback).
  This was what `frank-medium` ran until 2026-09-20, when the board swap made
  collectives legal and radiance TP=2 beat it on every axis (95.8 vs 45.7 t/s
  decode, 0.21 vs 0.72 s TTFT, 259 vs 72 t/s burst-4) — see
  [[frank-medium-radiance-mxfp4]]; the llama shape is the rollback target
  [[frank-medium-llamacpp-dflash2]].
- **Layer split is also the cross-vendor answer — and `-ot` is what cannot be
  combined with it.** On Dankclaw (2026-09-16) a CUDA 3090 + Vulkan iGPU layer
  split ran a real two-device graph: `-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85`
  held 14,994 MiB on the 3090 and gave 27.2 tok/s with 268.5 t/s prefill at 64k
  depth (iGPU alone: 25.8 / 231.8). What does NOT work is per-tensor placement
  across devices: `--override-tensor <group>=<device>` sets
  `has_tensor_overrides()`, a REQUIRED term of the scheduler's
  `pipeline_parallel` flag (llama-context.cpp:433 — the only place in the tree
  that reads it), and that flag is what sets `n_copies` in
  `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a
  non-primary device has no backend and the graph aborts (`ggml-backend.cpp:941`,
  reported with the leaf's own op name `(NONE)`). `=CPU` is the only override
  target that works, because the CPU backend is last in the list. Pure Vulkan
  changes nothing here — the gate is the override, not the driver mix. See
  [[x4-pair-experiment]].
- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design
  constraint, not a detail.** `pcie-link-test.cu` on Dankclaw (2026-09-16,
  services stopped): pinned H2D 6.72 GB/s, D2H 6.60 GB/s — 85 % of the Gen4 x4
  ceiling — with async no better than sync and pageable within 1 % of pinned,
  i.e. the copies are link-limited rather than staging-limited (the usual
  "pageable costs 2x" rule does not apply here). The card is x16-capable and the
  Strix Halo port beneath it is x8-capable (`LnkCap: 16GT/s, Width x8` on
  `00:03.1`), so the x4 lives in the adapter or cable; the link idles at
  2.5 GT/s x4 and trains up under load. Consequence for any engine that
  *streams* weights across it: a 3 MiB slab costs 474 us, and a 111 GiB A3B
  model needs ~1.70 GiB of active experts per token ⇒ **3.7 tok/s** (4.6 with a
  20 % resident cache), against **20.46 tok/s** for the same model with the
  experts *computed* in host RAM by llama.cpp (`-ot exps=CPU`, whence the link
  carries activations only) and 25–26.5 on the iGPU. Streaming over PCIe loses;
  computing in RAM does not care. FreeToken — expert LRU in host RAM, exactly
  this streaming shape — also loads only modelopt block-FP8 safetensors for
  `qwen3_5_moe`, so a Q4_K_M GGUF is unloadable there before bandwidth is even
  the question. See [[x4-pair-experiment]].


**ROCm / GPU**

- **Layer split needs no collective** — layers go in order and only the boundary
  activation crosses PCIe, so it is the fallback wherever a collective cannot be
  dispatched (and on an RCCL-less fork it beats tensor mode, which falls back to
  a butterfly allreduce). Pin the spec-decode drafter to one device: an unpinned
  draft under tensor mode aborts the KV build, and on a split its home is the
  card holding the target's last layers. Deployed history:
  [[frank-medium-llamacpp-dflash2]].
- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a
  trap.** With a missing `DST`, cp creates `DST` as a copy of `SRC` and drops
  one directory level. rocBLAS and hipBLASLt look for their device code at a
  fixed depth (`../lib/rocblas/library`, `../lib/hipblaslt/library/<arch>`),
  and the flattened copy fails at *request* time, not load time: the engine
  reads the weights, answers `/health` and dies on the first completion with
  `Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch
  : gfx1201`. Declare the destination, then copy the contents.
- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the
  ~50 GiB PLE table into anonymous memory → silent cgroup kill, dmesg is the
  only evidence. mmap pages ARE page cache. Never pin the PLE tensor with
  `-ot` (23 → 13.4 t/s).
- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve
  scripts export env files with `set -a`, so a tunable named `TEMP` leaks
  into HIP (`TEMP=1.0` sent the JIT cache to path `1.0` and segfaulted every
  load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a tunable
  these; check with
  `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"`.
- **`-DGGML_CUDA_FA_ALL_QUANTS=ON` is required** for q8_0 KV + flash
  attention on CUDA builds; without it prefill collapses ~12x (95 vs 1300+
  t/s, measured 2026-08-17).
- **Flash-Next MTP heads do not load on any current loader** — they need full
  per-layer `blk.<il>.hc_attn_norm.weight` and ship only `blk.48` (verified
  3x). Don't re-attempt without new loader support.
- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on
  qwen4exp needs `scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on
  the agentionai FAST layer set; the kingjones STRIX set works unpatched.
- **GPU budget on Dankclaw is the GTT aperture (~114 GiB), not the
  rocm-smi "dedicated" figure.** The Radeon 8060S (gfx1151) reports
  116,732 MiB (`llama-server --list-devices`;
  `gtt_total=122402963456` in `/sys/class/drm/card1/device/`). The older
  "64 GiB, SVM-backed" lesson was already stale — the CIRU Orca predecessor
  held 92.6 GiB of weights on the iGPU. `free`/`dmidecode` show the
  CPU-side carve-out — intentional, do not change.

**Load / crash diagnosis**

- Model load is slow (minutes cold); health returns 503 while loading; do
  not kill the process. `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). On
  halogen, the FIRST request after a (re)start can go silent ~45–60 s
  (health `engine_unresponsive`, 503s) and then self-recovers — do not
  restart over it.
- Crash-loop procedure: (1) `journalctl -u <svc> -n 100` + kernel line via
  `journalctl -k` (segfault at address 8 in `libamdhip64` = NULL unref in
  HIP); (2) run serve script, then raw binary — different outcome isolates
  the env file; (3) `gdb -batch -ex run -ex "bt 25" --args bash
  scripts/serve-<svc>.sh` — llama-server buffers stdout and loses the log
  tail on SIGSEGV; trust gdb, not the log; (4) `amdgpu: SVM mapping failed`
  spam is a symptom, not a root cause.
- Journal signatures: client abort logs `W srv stop: cancel task, id_task =
  N`; natural stop logs only `slot release` with `truncated = 0`. The server
  never preempts a running generation; excess tasks queue when slots are
  busy.

**Host memory / swap**

- Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff`
  of the old one, or live pages must land in RAM that `MemAvailable`
  overstates. Dankclaw swap (64 GiB at `/swap.img`, fstab-driven) protects
  the co-tenants, not frank-dragoon — pinned weights don't swap, and paging
  the KV pool is worse than the OOM it prevents.
- Host swap state is invisible to `deploy.sh` (no repo file) — check
  `swapon --show` before reasoning about headroom.
- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults
  to half of RAM, so the manual raise to 200 GiB (2026-10-05) vanished with the
  2026-10-12 23:17 POWER FAILURE on dankserver: the cap fell to 125,988 MiB,
  frank-archon's pinned-ledger preflight (floor 150,000 MiB) refused the start,
  and `Restart=always` re-ran that refusal **6,919 times in ~20 h** — a unit in
  `activating (auto-restart)` that exits 1 in 50 ms with no container and no
  engine log is a PREFLIGHT refusal, not an engine fault. Repo law now:
  `scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) raises the knob
  to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only
  after a write the kernel rejected. Never prescribe a manual sysfs write as the
  recovery procedure for a service that can boot before a human reaches it.
- **Silent RAM corruption outranks every software theory.** frank-archon's
  ~40k-token output corruption (2026-10-04/05, repetition/garbage in long
  agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
  by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and
  a short in-band pattern check (60 s, 8 GiB of 246 GiB) is not a stability
  proof — combined, those two instruments said "ruled out" while the RAM
  was the cause. "4800 is the JEDEC floor on 5600-rated DIMMs" is a rating,
  not a margin guarantee. When the fault is data-shaped (NaN states,
  garbage tokens) and a software revert doesn't move it, price in the RAM
  before the runtime — and remember a revert tested under the same marginal
  hardware is itself confounded. See the runbook's 2026-10-05 section.

**Runtime-specific guards**

- **Rootless podman cannot RAISE rlimits, and `--ulimit nofile=1048576`
  dies at container create, not at runtime.** runc reports
  `error setting rlimit type 7: operation not permitted` (type 7 =
  RLIMIT_NOFILE) when the requested hard limit exceeds the host process's
  hard limit — on Dankclaw that ceiling is 524288. Inherited-equal limits
  are fine, so `memlock=-1:-1` works iff the unit grants
  `LimitMEMLOCK=infinity` (systemd is privileged; the value flows through
  podman to runc without any raise). Upstream halogen's own run lines pass
  NO `--ulimit` at all — copy vendor run lines before adding flags
  (measured 2026-10-05 on the frank-dragoon cutover).
- **Kairic Edge** (if ever re-deployed): the pinned CK commit needs
  `third_party/composable_kernel/include/ck/config.h` generated manually
  (untracked; `git clean` removes it), and its greedy-argmax fastpath rejects
  non-greedy requests and reasoning budgets. Re-test chat before production.
- **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch;
  stock binaries fail the draft load with a tensor-shape mismatch.
- **A GGUF's custom tensor type picks the runtime — read the histogram before
  promising a file to a loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is
  ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`); stock
  llama.cpp and gufo refuse it at load, so ROCmFPX was not a preference but
  the only door. Counting tensor types out of the GGUF KV section takes one
  python read and settles a runtime argument in minutes. The build identity
  matters as much as the fork: the 2026-08-25 ROCmFPX builds SIGSEGV in
  `ggml_compute_forward_set_rows` on this same file, so
  `scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA.
- **Speculative-decode flags move as a SET.** Handing the ROCmFPX fork
  `--model-draft` with no `--spec-type` (the naive way to build a "no MTP"
  bench arm) segfaulted at load, core dumped, no diagnostic (2026-10-02). A
  baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND
  `--spec-type`, or it measures a crash instead of the absence of drafting.
- **An orphaned scratch server taxes the whole host.** A bench child left
  alive by a client-side exception kept ~26 GiB of weights resident on a
  121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
  Record the child pid and kill it in a `finally`, then prove `free` came back
  before believing a number. Kill by pid — never `pkill -f llama-server`,
  which also hits frank-zealot.

**llama.cpp serving**

- **MTP is a property of the FILE, not of the flags.** The `-mtp` GGUF build
  carries the head as `blk.<n>.nextn.*` (`qwen35.nextn_predict_layers = 1`);
  `--spec-type draft-mtp` then sets `load_mtp`, and the draft context runs on
  the MAIN model's weights (`shares_model`), so there is no second copy of a
  12 GiB model in VRAM and no `-md` argument. The plain build has no MTP
  tensors at all, and pointing `--model-draft` at the file would load it twice.
- **What can be chained, what cannot, and what a chain costs on a hybrid.**
  `--spec-type a,b` builds every enabled implementation and chains drafting
  through them, in a fixed internal order (ngram types first, then the
  drafters — CLI order is irrelevant). The gate is `ctx_dft`: **draftless**
  types (the ngram family) are added unconditionally, while `draft-mtp`,
  `draft-dflash`, `draft-dspark` and `draft-eagle3` all require a draft
  CONTEXT. So a chain may hold one context-backed drafter plus any number of
  ngram types — `draft-dflash,ngram-mod` is legal and measured (2.6x the
  drafter alone on prompt-echo traffic, byte-identical to it on prose at the
  fleet's 48/64 window) — but `draft-mtp,draft-dflash` dies at load: *"context
  type MTP requested but model doesn't contain MTP layers"*, because MTP
  builds its context against whatever model the draft context holds. Two
  context-backed types cannot share one process, and MTP's head only loads
  when MTP **is** the draft model. **A LEGAL chain can still lose on a
  recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35 hybrid
  measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer
  logs `recurrent state rollback is not compatible with 'ngram-mod' -
  disabling rollback support`, and MTP verification depends on that rollback,
  so acceptance drops on ALL traffic even where the matcher fires. Measure
  chains per architecture, never by analogy.
- **A drafter's KV scales with the TARGET's context, and its window policy
  decides the price.** The draft context is built at `llama_n_ctx(ctx_tgt)` with
  no flag to cap it, so a drafter declaring NO sliding window caches the whole
  window on `--spec-draft-device` and can OOM the first draft graph capture
  AFTER `/health` has answered. Read `sliding_window` out of the drafter's GGUF
  header before sizing, and prefer a skewed `-ts` over a context cut. Worked
  example: [[frank-medium-llamacpp-dflash2]].
- **A draft context bills its own f16 KV over the WHOLE POOL, and on a
  single-card endpoint that squeeze can cost more than the drafter wins.** With
  `--parallel N` the drafter pays the pool, not the slot, plus its own compute
  buffers. So the MTP question is never "does the file carry the head" but "what
  do draft KV + buffers cost against what the batch knobs leave for compute" —
  a per-shape answer. Measure the arm, never the flag; the booted arm table is
  on [[frank-zealot]].
- **A quant's scale overhead belongs in its byte rate; doubling a measured
  figure to "get the f16 number" invents 6.25 %.** q8_0 is 1.0625 B/element
  (32 codes + one 2-byte scale per 32-block), f16 is exactly 2.0 ⇒ the ratio is
  **1.882×**, not 2×. Derive byte rates from element count × block layout, then
  check them against the boot ledger — this repo carried an invented 2× number
  for weeks. Current KV math: [[frank-zealot]].
- **`--ubatch-size` has an OPTIMUM, not a plateau, and the curve is
  BACKEND-SPECIFIC — never carry it across hosts.** Its compute buffers land
  about TWICE as heavy on the card that holds the output layers and the drafter,
  so a split balanced at one batch is unbalanced at another: re-measure the
  balance after any batch change. The logical `--batch-size` is nearly free.
  A large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit
  — no totals model can see that. Ladders: [[frank-medium-llamacpp-dflash2]]
  (R9700 split) and [[frank-zealot]] (CUDA 3090).
- **On the CUDA 3090 that same knob has no interior optimum up to 2048.** A
  prompt shorter than the ubatch cannot see the knob at all — that is the
  control that proves the knob, not the traffic, moved the number. When a bench
  IS ordered, A/B it by restarting the SERVICE through a transient
  `FRANK_ZEALOT_ENV_FILE` drop-in, never a scratch port: one card holds one
  server. Numbers: [[frank-zealot]].
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
  size** (a DeltaNet boundary and a full-attention boundary can differ by ~1 GiB
  each with its own KV), so balance by measurement. An affine fit reproduces the
  deployed point and overstates the gaps in between.
  [[frank-medium-llamacpp-dflash2]].
- **ngram-mod prints no startup banner.** The load log shows the drafter's
  `adding speculative implementation 'draft-dflash'` and nothing at all for the
  matcher, so a chain can look unengaged while it is working. Identify it
  behaviourally (`draft_n` > 0 only where text repeats) or by the per-task
  `draft acceptance = ... mean len = ...` line. It costs no VRAM (a 64 MiB
  host-RAM hash table) and its window decides which traffic it can see: too
  short and it proposes on ordinary prose fragments, which costs verification
  without acceptance.
- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the
  per-slot number is the contract — not `ctx-size`.** `n_ctx_slot =
  CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT with
  a 400 even when the pool has room, so `models.yml` `contextWindow` must follow
  every reslice. Both slot directions cost or buy window, never pool; shorter
  slots decode slightly faster (less attention to scan, one fewer recurrent
  state). Measured reslices: [[frank-zealot]].
- **Price a VRAM budget from two boots, never from a formula — and an mmproj
  costs ~1.5× its file size** (the encoder's activation and compute buffers ride
  with it). Subtracting two booted rows isolates it exactly; extrapolating
  weights+KV+"a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7.
  Then solve `KV ≤ card − non-KV − headroom` and round to a multiple of
  `PARALLEL × 4096`. The two-boot table: [[frank-zealot]].
- **`images` and `image_url` are NOT interchangeable: only the content part
  reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
  model still invents an answer if the client used the legacy top-level
  `images` field on `/v1/chat/completions` — frank-zealot answered "Brown." for
  a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
  failure is silent because a hallucinated color is a plausible answer. Probe
  vision with a solid color and expect an exact word.
- **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
  `500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
  then served frank-dragoon: every tool-using agent on that endpoint was broken
  and the failure read like a model fault. Mainline llama.cpp 0.4.1 defaults
  `--jinja` on and the serve script pins it; halogen also accepts `tools` and
  returns `tool_calls`. Check tool support before routing a role at an endpoint.
- **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
  option and the same ~12x prefill collapse described in the ROCm note above.
  f16 KV needs nothing; frank-zealot ran f16 for that reason until
  2026-09-17, when `build-cuda` was rebuilt with the option ON so the
  endpoint could take q8_0 KV.
- **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both
  `-dev` and `-ts` follow this: `-dev CUDA0,Vulkan1` is two *single-device*
  configurations, never one two-device run, and `-ts 0.85,0.15` is two sweep
  values each becoming `[0.85, 0, …]` / `[0.15, 0, …]`, i.e. device 0 takes
  everything. Write `-dev A/B -ts x/y`. This artifact produced a wrong fleet
  conclusion on 2026-09-16 — "CUDA + Vulkan cannot serve one model", with two
  bogus OOMs that were really "all 111 GiB on the 24 GiB card" — and the bench
  table's own `dev` column was printing the truth the whole time. Any
  placement-dependent number needs an independent check: sample the other
  device's memory (`nvidia-smi`) while the run is loaded.
- **A model's own template can REJECT the harness's thinking level.** Qwen3.8's
  template accepts only `xhigh`, `medium` and `low` for `reasoning_effort` and
  raises on anything else — HTTP 500, not a clamp — so the harness's levels
  must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`).
  A role whose level is unmapped fails on its FIRST request, and the failure
  names Jinja, not the model: every subagent on `frank-zealot` died with
  `Unexpected reasoning effort high` until `high: xhigh` was added on
  2026-09-16. Halogen accepts the wider set, so the two entries legitimately
  differ.
- **The vision projector is NOT billed in llama.cpp's own fit check.** The
  engine prints `[mtmd] estimated worst-case memory usage of mmproj is
  873.10 MiB` and then fits WITHOUT it: frank-zealot's Bonsai-2 boot projected
  21,439 MiB and settled at 22,664 (2026-10-17). Price a 3090-class shape from
  settled `nvidia-smi` after a real boot, never from
  `model + context + compute` — the projector (and any drafter context) lands
  after `/health` would already have answered. Same failure class as the
  DSpark drafter OOM.
- **The FIRST request after a boot is a warmup artifact.** Measured on
  Bonsai-2/3090: 2.6 t/s prefill and 37.7 t/s decode on request #1, then
  1,258 t/s prefill and 54.7 t/s (per stream, two concurrent) on the next — a
  3× spread on the same shape. Benchmark the second request or later, or the
  graph-capture cost reads as a model fault.
- **Thinking tokens count against `max_tokens`, so a short-budget vision probe
  reads as blindness.** The Bonsai-2/`image_url` probe at `max_tokens: 64`
  returned `content: ""` with `finish_reason: "length"` and half a thought in
  `reasoning_content` — the image HAD reached the model (322 prompt tokens vs
  42 for the text alone). At 512 tokens the same image answers "Red". Budget
  ≥512 for any thinking model you probe for vision.
- **Quantized KV is not the only dtype law a fork build can hide.** Vendor
  builds carry their own compile options: frank-zealot's PrismML prebuilt
  served `--cache-type-v q8_0` under flash attention at the exact 34,816
  B/token (the `GGML_CUDA_FA_ALL_QUANTS` door), so a build decision must be
  proven by the KV buffer line in the boot ledger, not by the build flags you
  remember passing.
- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed.**
  Without it the engine answers **501 Not Implemented** on `/metrics`, so the scrape
  fails while `/health` answers 200 and `LlmEndpointDown` fires beside a healthy
  service — that is how `frank-zealot` sat red across four payload generations
  (the oldest retained sample of `up{job="frank-zealot"}`, 2026-10-11, is already
  0). halogen serves `/metrics` natively, so the asymmetry reads as "only the 3090
  is broken". Diagnose with `lastError` from `/api/v1/targets`, not the alert list,
  and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an
  alert that has been firing for weeks.
- **A leftover bench drop-in silently hijacks the next deploy, and no health
  gate sees it.** The sanctioned bench-window method — a transient
  `Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env` drop-in, the arm env
  `source`ing the repo env and then overriding it — outlives the bench unless
  it is retired. On 2026-10-18 a leftover `frank-zealot.service.d/ptq1.conf` —
  the arm Frank had ordered at 02:36 and measured from 03:14 — made the cutover
  deploy boot a **different model at a different shape** (3 slots of 133,888,
  not 3 × 262,144) while `/health` answered 200, `/v1/models` echoed the alias
  (a flag, not a fact) and the Playwright gate passed 33/33. After deploying any
  recently-benched LLM service: `systemctl show <svc> -p DropInPaths` (must be
  empty), then `tr '\0' ' ' </proc/$(pidof llama-server)/cmdline`, then the boot
  ledger — and retire drop-ins to `~/.AGENTTRASH/<date>-<topic>/` +
  `daemon-reload`.
- **A bench verdict that never reaches the repo is indistinguishable from a
  rogue arm.** Same window: PTQ1_0 served production for 24 minutes with real
  numbers (~48 t/s decode, 801 t/s prefill at 141k), but the figures lived only
  in `/tmp` env files and journald, so the next agent on the host read the
  surviving drop-in as an unfinished experiment and reported a hijack. Write the
  measurement into the runbook and the arm script's header in the session that
  takes it — `/tmp` and the journal are not records.

**Ops**

- Never `pkill -f` a pattern that appears in your own command line — it kills
  your ssh session. Use `pkill -x <exact-name>`.
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in
  committed scripts (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts
  without it.
- ComfyUI on Dankclaw runs with **GPU access** (`/dev/kfd` + `/dev/dri` via
  `AddDevice`, `GroupAdd=keep-groups`, plus the udev rule that makes those nodes
  world-writable — user-unit spawns carry no supplementary groups). It must NOT
  pass `--cpu`: the gfx1151 ROCm wheels exist and the unit's `--cpu` era ended
  2026-09-16, when a stale device-less container was found restarted 9,146
  times (`RuntimeError: No HIP GPUs are available`).
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its
  ports off the gluetun service.
- Dankclaw boots **`iommu=pt`** since 2026-10-18. The boot line is tracked in
  `dankclaw/host/grub` and moves ONLY through `scripts/set-dankclaw-bootline.sh`
  (diff, install, `update-grub`, verify `grub.cfg`, `--reboot`) — never by
  hand-editing `/etc/default/grub`. It replaced `amd_iommu=off` (2026-09-11),
  adopted on the vendor's 13–16 % prefill figure and never measured here: the
  local A/B on byte-identical traffic says identity mode costs **nothing**
  (+2.0 %, 1,616 t/s cold prefill), so the XDNA2 NPU returned for free —
  `amdxdna` binds `c6:00.1` (PASID width 10, identity group 26) and the old
  `err=-19` only ever asked for an enabled IOMMU. **AMD PerfOpt** (iommu tree
  2026-09-24, Linux 7.4) still skips the page walk outright for the privileged
  iGPU, but it arms only in the identity domain and 26.04 caps at `7.0.0-38`.
- Rootless Quadlet containers run with `--replace --rm`: `podman logs` covers
  only the CURRENT instance; history across recreations lives in
  `journalctl --user -u <name>.service`. Read rootless storage WITHOUT sudo
  (per-user storage — `sudo podman logs` finds nothing).
- **A dead rootless podman pause process breaks ALL container management on the
  host, and `podman system migrate` panics in 5.7.0.** Symptoms: every `podman`
  command fails with `invalid internal status … could not find any running
  process`, every container start fails binding podman's bridge addresses
  (`Error starting server failed to bind udp listener on 10.89.x.1:53: Cannot
  assign requested address`), and every `Restart=always` quadlet loops. Container
  storage is intact; what is gone is `/run/user/<uid>/libpod/tmp/alive` (a UUID
  token, not a pid) and the rootless netns holding the bridges — deleting the
  stale token is not enough, and `podman system migrate` segfaults in
  `teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28
  quadlets return on their own — check `Linger=yes` and `WantedBy=default.target`
  first). Seen 2026-09-16, hours into heavy GPU work on the same host — but the
  trigger is NOT established (`journalctl -k` recorded no OOM kill; the kernel
  had no other word on it). Treat the pause process as host state that can die
  for its own reasons, and reach for the reboot when it does.
- **A container cannot reach a `Network=host` service at the host's LAN IP.**
  netavark DNATs *published* ports only: from a bridge container
  `192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
  gateway `10.89.1.1` are both refused. Use `host.containers.internal`
  (169.254.1.2). This is what 502'd `hermes.franksdank.cloud` until the Caddyfile
  upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP
  was open from that bridge — it never was).
- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as
  `.partNofM` fragments.** `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF`
  carries Q4_K_M (37.00 + 37.00 + 36.97 = 110.97 GiB) and Q4_K_S (35.00 + 35.00
  + 34.09 = 104.09 GiB) as parts, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*,
  IQ4_XS) are single files. Fetch the parts, verify each against the Hub's LFS
  sha256, concatenate in numeric order — that is how the deployed 111 GiB file
  was assembled. Listing the Hub tree and filtering `endswith(".gguf")` makes
  Q4_K_S look like it does not exist.
- **A fetch script must `-f` and verify, or it reports success on an access
  error.** `curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body
  as the "artifact": a gated repo yielded three 153-byte shards with identical
  sha256 values and a cheerful `DOWNLOAD_DONE` in 20 seconds for 104 GiB. Use
  `-f`/`--fail-with-body`, compare each file's byte count to the Hub's declared
  size, and check the LFS sha256 — `scripts/download-frank-medium-gguf.sh`
  already does all three; ad-hoc fetches tend not to. Gated repos (e.g.
  `orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`, same 104.09 GiB Q4_K_S)
  return 403 even with the host's HF token until that account accepts the gate.
- **A scout verdict is a lead, never a finding.** In the 2026-10-19 vault de-dup
  campaign, four read-only scouts worked the same pages: one classified era pages as
  stale *because* they document retired payloads (that is an era page's content), one
  reported files that do not exist, one reported duplications that grep disproved
  (`457.9 t/s` and the ping-RTT table are solo-held by `Ethernet Bandwidth Math`),
  and one cited a line number past the end of the file it judged. Re-verify every
  claimed duplicate and every claimed staleness against the file and today's fleet
  state before editing; a scout yield buys no evidence.

## Runbook Index

One page per live endpoint; every payload swap it has ever run is a row in that
page's **Eras** table, with the rollback path in the same row. There is no
per-attempt page and no per-swap ledger anywhere else.

| Service | Page |
|---------|------|
| `frank-archon` — radiance, 2× R9700 TP=2, Dankserver | `serverAppConfigs-docs/LLM/frank-archon.md` |
| `frank-dragoon` — halogen flash-server, Dankclaw iGPU | `serverAppConfigs-docs/LLM/frank-dragoon.md` |
| `frank-zealot` — llama.cpp (PrismML fork), RTX 3090 | `serverAppConfigs-docs/LLM/frank-zealot.md` |
| `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
| Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |

Retired **endpoints** (not payloads) keep their own pages — `frank-medium`,
`frank-npu`, the pre-name Dankclaw/Dankserver eras — because no live page owns
them; `serverAppConfigs-docs/index.md` is the locator for every page in the
vault. Retired scripts live in `old/` and `.AGENTTRASH/`.

## Why

Direct edits on the remote drift from version control, and future agents
won't know what changed. Git is the single source of truth. The two workloads
share two hosts, one Caddy, and one Gitea; one repo keeps them in one
history.

## Summary

| Context | Action |
|---------|--------|
| Editing a compose/Caddyfile/serve script | **Through git** — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote. |
| Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before editing. |
| Unstaged local changes | **Commit or discard** — check `git status` before starting work; ask if unsure. |
| Deploying a committed change | **`scripts/deploy.sh <target>`** — pushes, pulls ff-only, plans, dispatches, health-checks. |
| Container without a Quadlet unit | **Give it one** — boot-persistent, self-healing `.container` via git. |
| Changing a model/payload | **Swap it** — edit the trio, deploy, `health` + one chat/tool/vision probe. No benches unless Frank orders one; if it misbehaves, revert. |
| Writing a doc | **One owner per fact** — edit the endpoint/service page; everywhere else links. Never restate a number. |
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |