serverAppConfigs/AGENTS.md

AGENTS.md — serverAppConfigs Agent Rules

Overview

This repository is the single source of truth for the two homelab servers and the FrankBook-Pro Mac mini. It tracks the Podman container stacks and the LLM inference fleet. One repo, one checkout per host, one deploy script. All services run from /home/dank/serverAppConfigs on each host (the 2026-08-22 merge absorbed the former vllmScripts repo, history preserved).

• Dankserver (192.168.0.235) — media/library stack (RomM, Copyparty, Calibre-Web, Jellyfin, Plex) plus the frank-archon LLM service.
• Dankclaw (192.168.0.143) — AI/search/dev/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, Alloy, Pi-hole) plus TWO LLM services: frank-dragoon (flash-server container on the gfx1151 iGPU) and frank-zealot (llama.cpp CUDA on the RTX 3090). Caddy on Dankclaw reverse-proxies both hosts.
• FrankBook-Pro (192.168.0.112) — macOS mini running the *arr stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside its podman machine; all egress rides the Gluetun WireGuard netns. Deployed via the frankbook-pro handler.

Fleet topology (ACTIVE)

Target	Service	Runtime	Payload	Port
Dankserver	frank-archon	radiance, digest-pinned, root unit, TP=2	publisher qwen3.8-next-flash-fp8-iq4r-moe.rad	7770
Dankclaw	frank-dragoon	halogen-flash-server 0.16.2, digest-pinned, root unit	qwen38-flash-next-v2-abliterated.hgn + n-gram + vision tower	7770
Dankclaw	frank-zealot	llama.cpp PrismML fork build 10777, root unit — the only loader for this payload	Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj	7771

Payload identity, shape, eras, laws and rollback live in ONE page each: frank-archon, frank-dragoon, frank-zealot. scripts/<service>.env is the authority for every number; a page that disagrees with it is the bug. A retired payload is a row in that page's Eras table — there is no page per attempt and no separate swap-history file.

Repository Layout

• dankserver/ — compose.yaml + Caddyfile + .env.example for the media stack
• dankclaw/ — compose.yaml + Caddyfile + .env.example for the AI stack
• shared/ — homepage, Prometheus, Loki, Alloy, SearXNG configs
• scripts/ — deploy.sh / deploy-plan.py / deploy-remote.sh, LLM serve scripts, systemd units, env tunables, bench harnesses
• old/ — retired LLM scripts (reference only)
• serverAppConfigs-docs/ — knowledge base (Obsidian vault, OKF format). The managed skill library is distilled there one page per skill-cluster: LLM/index.md → Cross-Cutting References, plus Operations/index.md, Agent/index.md, and the Infrastructure/Dankclaw/FrankBook-Pro service pages. There is no log.md: the change diary was destroyed 2026-10-19
• tests/ — bats suite + Playwright e2e

Git Workflow — trunk-based (fleet law, aligned 2026-10-18)

All edits flow through git. Never edit files on a remote machine directly.

main is the only long-lived branch: no develop, no release branches, no back-merges. Frank is the only dev; this repo and dank/command-center follow the same regime.

1. Branch fresh — git fetch, then branch off the current origin/main (<type>/<slug>, type ∈ feat|fix|docs|chore). If main moves under you, merge it into your branch — never rebase.
2. Source the live config from the target machine
3. Commit the fetched version here (redact secrets into .env)
4. Edit locally
5. Test — ./tests/run.sh must pass
6. Push the branch and open a PR against main — origin is Gitea on Dankclaw. Merge with --no-ff; never merge a PR without an explicit, current OK from Frank. The moment the PR is open, open it in Frank's browser (xdg-open <PR-URL> — a Firefox tab on the workstation); never make him hunt for it in Gitea. Releases: tag vX.Y.Z straight on main.
7. Pull on the target (deploy.sh does this)
8. Deploy — scripts/deploy.sh plans and dispatches only changed services

Attribution footer (issues and PRs) — every issue or PR body you open MUST end with:

---
cwd: <working directory> · session: <omp session id>

Session id from /status; cwd is that session's own working directory — not the build checkout, when they differ (note the build worktree in the body instead). Machine-made artifacts stay attributable to the box and session that spawned them.

Deployment

./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankserver
./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-archon
./scripts/deploy.sh dankclaw --dry-run

The planner maps changed paths to services; the remote runner restarts only the selected service and records its revision outside the checkout. Documentation-only changes are a no-op. LLM services are selected only by their own files — every endpoint (frank-dragoon/frank-zealot/frank-archon) by its unit/env/serve trio in scripts/ — so unrelated container changes cannot interrupt inference. Set FRANK_SUDO_PW for headless runs on a host without passwordless sudo; the value never enters the repo.

deploy.sh <target> with no --service is a fleet-wide DRAIN, not a sync. It also walks every per-service revision marker under the deploy-state dir and restarts each service whose marker is behind HEAD and whose paths changed since that marker — so a backlog accumulated by service-scoped deploys fires all at once, and one failing health gate aborts the run with every later service left pending. Name the service unless a drain is what you want (./scripts/deploy.sh dankserver --service frank-archon).

Deployment completeness (MANDATORY)

A deploy is not complete when deploy.sh exits green. It requires:

1. ./tests/run.sh — all seams green, and
2. the Playwright e2e suite (tests/e2e/run.sh) — runs automatically as the [3/3] gate at the end of deploy.sh against the LIVE fleet; a behavior-affecting deploy that fails it exits DEPLOY INCOMPLETE. Bypass only with --skip-e2e in an emergency. The gate runs the conditional frankbook Playwright project (the *arr/qBittorrent walk, tests/e2e/specs/19-frankbook.spec.ts) ONLY when a frankbook blast-radius path changed since origin/main — that spec, frankbook-pro/**, shared/homepage/services.json, or the dankclaw/Caddyfile serving the arr vhosts — so a sleeping FrankBook-Pro never fails an unrelated deploy. The default project set excludes it everywhere else (E2E_INCLUDE_FRANKBOOK=1 opts in; see serverAppConfigs-docs/Architecture/E2E Suite.md).

graph LR
    A[Edit locally] --> B[Tests pass] --> C[Commit + push]
    C --> D[deploy.sh plan] --> E[Target pulls ff-only]
    E --> F[Dispatch selected service] --> G[Per-service health]
    G --> H[Playwright e2e gate] --> I{green?}
    I -->|yes| J[Deploy complete]
    I -->|no| K[DEPLOY INCOMPLETE]

Boot Persistence & Self-Healing (MANDATORY)

Every deployed, working Podman container MUST have a self-healing, boot-persistent Quadlet unit. A container that runs only from compose.yaml or a manual podman run dies silently at reboot — rootless Podman does not auto-start bare containers, and restart: unless-stopped does not survive the explicit stop that shutdown performs. Never report such a service as "up".

Unit requirements:

• Boot persistence — [Install] WantedBy=default.target, installed to ~/.config/containers/systemd/<name>.container, enabled in the user manager, with linger on for the deploy user (loginctl show-user dank -p Linger → Linger=yes).
• Self-healing — Restart=always (or on-failure) with a short RestartSec; quadlet health check where the service has one.
• Repo provenance — the .container file lives in this repo (<host>/<svc>/quadlet/), is installed by the deployer, and is covered by the quadlet test seam. The host-side unit is generated state; the repo file is the source of truth.

Before declaring a service deployed: systemctl --user is-enabled <name> → enabled, is-active → active, and the container answers its health check. A running container with no unit is a gap: write the unit, test it, deploy it.

This rule exists because reboots found gaps twice (2026-08-26: Caddy as a bare compose container never returned; 2026-09-11: five observability services had no units). The observability services now have units under dankclaw/observability/quadlet/ and their compose definitions were removed — one owner per container name. The stack is configured and delivering as of 2026-09-17: a Grafana datasource and the llm-fleet dashboard are provisioned from shared/grafana/provisioning/, shared/prometheus/rules/ llm-fleet.yml loads 5 alert rules + 1 recording rule, and firing rules are delivered: Prometheus → Alertmanager (dankclaw/observability/quadlet/ alertmanager.container, loopback :9093) → host-side Signal relay (scripts/signal-relay.py, user unit, 192.168.0.143:18093 — the host LAN IP, the one bind a container can reach as host.containers.internal; iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311. Prometheus targets read down for structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is a blackbox exporter. frank-zealot was down for a different reason until 2026-10-18: llama.cpp will not serve /metrics at all without --metrics. See serverAppConfigs-docs/Infrastructure/Observability.md.

Testing

Tests are mandatory before commit (bats required on the dev machine):

./tests/run.sh

Seams: compose validity, env consistency (no leaked secrets), Caddyfile validity, quadlet units, docs completeness including vault link integrity, script syntax (shell + python), LLM unit integrity, LLM secret hygiene, deployment flow, service-scoped planning, homepage config/pages/deploy/checker, and e2e_scaffold — which fails if a Playwright spec file's number prefix is in no project, because such a spec silently never runs. See serverAppConfigs-docs/Architecture/Testing Policy.md.

Secret Rules

• Never commit real secrets. The live Gitea DB password and sudo password are test-enforced leak checks.
• .env files are git-ignored; commit only .env.example with placeholders. Exception: scripts/*.env LLM tunables (paths and numbers, never credentials).
• Do not document passwords anywhere in the repo. SSH keys are the auth mechanism; FRANK_SUDO_PW covers headless sudo and never enters the repo.
• A token passed as curl -H "Authorization: Bearer …" is world-readable. It lands in /proc/<pid>/cmdline, so any ps/pgrep -a on the host prints it — that leaked an HF token into a session transcript on 2026-09-16, while merely checking whether a download was still running. Pass secrets out of band (curl -K <mode-600 file> carrying the header, or HF_TOKEN in the environment for hf/hf_transfer) and never pgrep -a a fetch command that carries an auth header.

External Exposure Policy (MANDATORY)

Every Caddy-proxied service is either behind the auth chain (Anubis :8923 → Authentik embedded outpost) or explicitly internal-only — no public DNS name, no vhost. New services default to internal-only until guarded.

When guarding a new vhost — provider first, then route:

1. Create the outpost proxy provider FIRST (external_host = exact vhost URL with trailing slash; internal_host = http://caddy:8017). If the vhost flips first, authenticated traffic hits the outpost error page.
2. Flip the vhost to reverse_proxy anubis:8923 with header_up X-Real-Ip {remote_host}, and add the post-auth @<svc> host <vhost> handler on the http://:8017 listener routing to the real upstream.
3. Anubis guard set, Caddy TLS edge, and outpost provider list change together in one deploy.
4. Service-to-service traffic targets internal addresses (loki:3100), never the public hostname — non-browser clients cannot solve challenges.

Exempt categories (document the reason in the vhost comment): the IdP itself (auth.), git smart-HTTP endpoints, services with their own complete auth.

Deletion & Trash Policy (MANDATORY)

Never rm. Move deletions into trash so they stay recoverable:

1. In-repo files → .AGENTTRASH/ beside the item's directory.
2. Elsewhere → dated staging dir ~/.AGENTTRASH/YYYYMMDD-topic.
3. List staged items and sizes; wait for explicit user approval to purge.

rm, rm -rf, and git clean -f are forbidden without explicit permission. git * --no-verify is forbidden. The trash is a temporary hold, not an archive.

Model Swaps (fleet law, 2026-10-18)

Swap the model. Do not instrument it. A payload change is: edit the trio (env / serve script / unit) → scripts/deploy.sh <host> --service <svc> → /health answers → one chat turn, one tool call, and one vision probe on an endpoint that serves images. That is the entire procedure.

• No benchmarking unless Frank orders it. No pre-swap baseline battery, no A/B arms, no decode/prefill ladders, no acceptance tables, no numbers written into any page afterwards. Data exists only when he asks for it, and then only in the file he names.
• It doesn't work → roll back. Revert the commit, redeploy, done. No post-mortem page, no investigation log, no trial-drop-in forensic tour. If the fault is not obvious, one line in the endpoint page's Laws section is the whole record.
• What still gates a change: ./tests/run.sh before commit, the deployer's per-service health gate, and the Playwright e2e gate. Those are the fleet's evidence. A bench is not evidence of correctness and buys nothing a swap needs.

When a bench IS ordered

STOP the production service first — these servers serve real users, and a bench beside a live service starves the GPU and corrupts the result. That covers any GPU experiment, not just benchmarks, including a bare 2-rank RCCL/torch.distributed probe: on 2026-09-16 one was run against a live service "because it only needs 200 MB of VRAM" and the host went hard down ~20 minutes later (no ARP reply, 100 % ICMP loss, power cycle required; a hung AQL dispatch wedging the driver is the plausible mechanism). Telling an off machine from a wedged one: ip neigh show <ip> from a peer — FAILED means off or NIC dead; a merely wedged kernel usually still answers ARP.

1. Stop and mask (self-healing would restart it): sudo systemctl stop frank-dragoon frank-zealot on Dankclaw, sudo systemctl stop frank-archon on Dankserver, then mask each. mask FAILS on a relinked unit (the file in /etc/systemd/system/ is itself a symlink into the checkout); stop alone is enough for the window, and the deploy restores the unit — re-deploy after benching rather than unmasking.
2. Bench against the service itself. No scratch ports on an LLM host — one process owns the card. Use the runtime's own harness (llama-bench; a live llama.cpp server also reports honest timings per request, including draft_n/draft_n_accepted — but on frank-zealot's PrismML fork build 10777 that object only reaches you on /v1/chat/completions: a greedy ignore_eos run on the raw /completion path there answers 500 "The model produced output that does not match the expected Content-only format" (measured 2026-10-18). halogen has no /completion path at all, so its rates are wall-clock around /v1/chat/completions plus the boot ledger. Generate ≥128 tokens: at 8, inter_token is first-token noise and run-to-run spread swamps every real difference.
3. Unmask, start, verify curl http://192.168.0.<host>:7770/health. Retire every bench drop-in the same session (systemctl show <svc> -p DropInPaths must come back empty) and write the numbers into the file Frank named — /tmp and the journal are not records.

Instrument traps that survive from the era when this was routine: gufo bench cannot rank speculative decode and inverts the ranking (its corpus reports acceptance=0.933 ± 0.000 with an identical trace sha256 across draft widths — the same predictable text every run); llama.cpp's Server-Timing carries the unit in the name half (ttft;dur=402.485), so splitting on = keys the dict ttft;dur and every lookup silently returns nothing; halogen 0.16.2 answers /completion, /ping, /props and /tokenize with 404 and sends no Server-Timing; and the first request after a boot is a warmup artifact (measured 2.6 t/s prefill then 1,258 t/s on the next), so never report request #1.

CRITICAL: never systemctl disable a system unit on these hosts — it deletes the unit file from /etc/systemd/system/, not just the symlink. mask symlinks to /dev/null and is safe/reversible.

Documentation Policy (fleet law, 2026-10-19)

Documentation exists for point of use. A page earns its bytes only if it helps whoever performs the next task — the shape to deploy, the flag that must be passed, the failure signature to recognise, the rollback to run. It is not a lab notebook and not a ledger. Losing information is an acceptable, usually correct outcome. Never let an orphan census ("this is the only copy of that fact") decide what to keep: sole custody of a useless fact is still useless.

• The purpose test — before you write, and again before you keep: would the reader act differently because of this line? No → delete it.
• Measurements are disposable. A rate or throughput figure (t/s, tok/s, ms, acceptance %, a bench ladder, a measured MiB) for a payload, runtime, or layout that is not running now and not being compared now is destroyed, not filed — not in an Eras row, not on a technique page, not "preserved for history". AI moves faster than this vault: the number is wrong before anyone reads it, and a wrong number in a runbook gets believed. Keep the law the measurement taught; drop the measurement.
• One fact, one owner, every other file links with [[Page]]. The env file beside the serve script is the authority for every live number; the endpoint page is the authority for the shape and the wire contract.
• One page per endpoint: current shape, Eras (window | payload | runtime | replaced because | rollback — the rollback is a commit, not a number), the Laws, the wire contract, the files that own it. A swap edits the shape block and appends one row; the retired era's own page is destroyed to .AGENTTRASH/, never merged into anything.
• log.md is retired (2026-10-19: 2,380 lines destroyed). Git holds the chronology. A causal law belongs to the page that owns the thing it governs, or to Key Lessons Learned below if it is cross-cutting — never to a diary.
• Index pages are locators: one row, one clause, no numbers.

Model Config (harness)

The harness reaches the fleet through .omp/agent/models.yml in the operator's home — /home/blank/.omp/agent/models.yml on the dev box, which is where the harness runs; there is no copy on the servers. Each provider is one HOST:PORT; each model id is the served alias. Official model ids are the StarCraft unit names (2026-10-08), on the wire since 2026-10-09, and since 2026-10-10 they are also the SERVICE names: frank-archon → 192.168.0.235:7770 (FRANK_ARCHON_SERVED_NAME), frank-dragoon → 192.168.0.143:7770 (HALOGEN_MODEL_ID), frank-zealot → 192.168.0.143:7771 (ALIAS). The 2026-10-10 cutover renamed the units, scripts, env files, containers, deploy services and runbooks to match — the three former service names are fully retired and exist nowhere in the repo; historical text in runbooks uses the current names. Verify a rename on /v1/models, never on a chat response: radiance echoes the requested model name, so a 200 proves nothing about the served id (llama.cpp and halogen ignore it).

Resolution rules: the default model resolves as a defaultProvider + defaultModel pair (the provider must serve the model); a subagent override uses a bare model id, which resolves only when unique across providers. Two ports on one host need two providers — the file cannot express "same host, different port".

When renaming a service, retiring an endpoint, or moving a model: update the provider block, model id, contextWindow, maxTokens, input in models.yml, then fix every modelRoles entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As of 2026-10-06 all three endpoints accept images (input: [text, image]) — frank-dragoon returned to vision with the flash-server v2 swap (the v2 vision tower loads beside the checkpoint; the text-only halogen 27B window closed 2026-10-06) — while frank-archon takes them natively (radiance runs the checkpoint's 27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary are the engine core; --mm-max-patches available), and frank-zealot through the publisher's own Q8_0 mmproj passed as --mmproj — Q8_0 is the publisher's requirement for concurrent image requests (the card states the BF16 tower can crash llama-server when several image requests run at once, and this endpoint runs several slots). frank-zealot's contextWindow is 200704 — the PER-SLOT window (--ctx-size 602112 --parallel 3 since the 2026-10-19 reslice that funded a third slot AND an armed MTP drafter), deliberately below the model's native window; llama.cpp answers a longer prompt with a 400 naming the limit, never a truncation, so the harness must compact before it. Note also that the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3 parser, live-verified) emit reasoning_content, vLLM emits reasoning — models.yml sets reasoningContentField per endpoint to match. There is no audio input anywhere in the fleet: that capability left with the FLM NPU endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's audio encoder, and left again with the Gemma4 retirement on 2026-10-01 (neither the MiMo nor the Qwen3.5 tower carries audio).

Each endpoint has ONE page in the vault: shape, Eras, Laws, wire contract, rollback. The numbers below are only what a consumer or a deploy gate needs; the env file beside the serve script is the authority, and a page that disagrees with it is the bug.

frank-dragoon — halogen flash-server, Dankclaw iGPU, :7770

• Owns: scripts/frank-dragoon.env, scripts/serve-frank-dragoon.sh, scripts/frank-dragoon.service, container frank-dragoon-halogen, image digest-pinned ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…. Engine stdout (the boot ledger: weights, KV pool, clamp) goes to /home/dank/logs/frank_dragoon.log, NOT the journal.
• Shape: HALOGEN_CTX=200704 is the per-request ceiling (a longer prompt is a 400, never a truncation); HALOGEN_KV_SLOTS=2 over ONE HALOGEN_KV_POOL_POSITIONS=401408 pool — pool positions buy concurrency, the per-request window never moves; HALOGEN_SPEC_ADAPT=0 keeps the MTP drafter always armed (the adaptive default disarms it and halves decode); server sampler 1.0/0.95/20.
• Harness contract: models.yml contextWindow: 155648 = ceiling − 45,056. The clamp is min(45056, window − est − 4096) and the chars/4 estimate under-counts, so a harness window equal to the ceiling makes every long request a guaranteed 400. Reasoning arrives in reasoning_content; the template rejects reasoning_effort outside xhigh|medium|low; images are OpenAI image_url parts (probe ≥256 px); response_format unsupported; /completion, /ping, /props, /tokenize are 404.
• Page: frank-dragoon.

frank-zealot — llama.cpp (PrismML fork), RTX 3090, :7771

• Owns: scripts/frank-zealot.env, scripts/serve-frank-zealot.sh, scripts/frank-zealot.service; fork build llama-prism-b10777 is the ONLY loader (the payload carries the fork's custom PQ2_0 tensor type plus prism.hadamard.* dequant metadata — stock llama.cpp refuses it, and a loader ignoring that metadata dequantizes wrong); payload Bonsai2-9B-MTP/…PQ2_0.gguf + the publisher's Q8_0 mmproj (the BF16 tower can crash the server with concurrent image requests).
• Shape: --ctx-size 602112 --parallel 3 = three conversations at 200,704 each, CACHE_TYPE_K=f16 + CACHE_TYPE_V=q8_0 (asymmetric: 25,088 B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024, SPEC_TYPE=draft-mtp (depth 2), --flash-attn on, --jinja, --metrics.
• Harness/gate contract: contextWindow: 200704 is the per-slot window (n_ctx_slot = CTX_SIZE / PARALLEL), and llama.cpp answers a longer prompt with a 400 rather than truncating. --metrics is mandatory or Prometheus reads 501 = DOWN beside a healthy service. Presence penalty has no GGUF metadata key, so it must reach argv. This endpoint is the fleet's compaction hub.
• Page: frank-zealot.

frank-archon — radiance, two R9700s TP=2, Dankserver :7770

• Owns: scripts/frank-archon.env, scripts/serve-frank-archon.sh, scripts/frank-archon.service, scripts/ttm-preflight.sh, template scripts/qwen38-froggeric-v22.5.jinja; image digest-pinned stilldeadcode/radiance@sha256:4c3f4845…; container name = unit name.
• Shape: TP=2 over wht6, --max-num-seqs 3, --max-model-len 200000, expert_tiered placement, 44,928 MiB pinned host pool per rank, VRAM pair --vram-weights-mib 25600 --vram-kv-mib 5120 (ONE budget: weights + kv ≤ 30,969 MiB/card), fp8 KV, MTP depth 2, MNBT 2048, prefix cache 8 GiB host
  • 64 GiB disk, n-gram table pinned to RAM.
• Harness/gate contract: models.yml contextWindow MUST quote 200,000 (over-length is refused, never truncated); the ~147 GiB pinned-RAM ledger is why this is a root unit, and ttm-preflight.sh raises ttm.pages_limit itself because the kernel resets that knob to half of RAM on every boot; /health binds only after the full load, so a bare health gate is honest.
• Page: frank-archon.

Key Lessons Learned

Durable operational lessons from the fleet's history. Full detail in the linked runbooks.

PCIe / multi-GPU

• RCCL collectives need PCIe atomics on every participating GPU, and the failure never names atomics. The kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch them on a device without atomics and the error surfaces as hipErrorIllegalState from hipLaunchKernel, not as anything that names atomics. The value is consumed at driver PROBE time, so writing the AtomicOp Requester bit with setpci at runtime changes nothing. Check the CHAIN: lspci -vv -s <gpu> and every upstream bridge must show AtomicOpsCap: Routing+ (a Routing- bridge is a hardware wall), and the endpoint's Device Control 2 (setpci -s <gpu> CAP_EXP+28.w) must have bit 6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves nothing: NCCL short-circuits at world_size=1. PCIe Link and Atomics
• When a GPU collective dies, read the runtime, not the framework. The message that identified the cause came from AMD_LOG_LEVEL=3 (Pcie atomics not enabled, hostcall not supported → AQL dispatch failed!), never from vLLM or RCCL — the env var costs one line and prints the failing HIP call with its return code. PCIe Link and Atomics

ROCm / GPU

• Bundling a ROCm runtime next to a llama.cpp binary: cp -r SRC DST/ is a trap. With a missing DST, cp creates DST as a copy of SRC and drops one directory level. rocBLAS and hipBLASLt look for their device code at a FIXED depth (../lib/rocblas/library, ../lib/hipblaslt/library/<arch>), so the flattened copy fails at REQUEST time, not load time: the engine reads the weights, answers /health and dies on the first completion with Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch : gfx1201. Declare the destination, then copy the contents.
• mmap must stay on for the big-PLE models: --no-mmap forces the ~50 GiB PLE table into anonymous memory → silent cgroup kill, with dmesg as the only evidence. mmap pages ARE page cache, and pinning the PLE tensor with -ot was slower than letting it page. Llama Cpp Serving Sizing
• HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE. Serve scripts export env files with set -a, so a tunable named TEMP leaks into HIP (TEMP=1.0 sent the JIT cache to path 1.0 and segfaulted every load, fixed 2026-08-13 by renaming to TEMPERATURE). Never name a tunable these; check with strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE".
• GGML_CUDA_FA_ALL_QUANTS is dead; do not build for it. Verified on this host 2026-10-19: master's ggml/cmake/common.cmake warns it is deprecated in favour of GGML_CUDA_FA_QUANTS, the successor's DEFAULT pair list (q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16) already compiles the quantized-KV kernels with the old flag OFF, and neither llama tree on this host references the old macro anywhere under ggml/src/ggml-cuda/. The old "~12× prefill collapse" belongs to the older source line. Prove a build's KV ability from its CMakeCache.txt. frank-zealot
• Flash-Next MTP heads do not load on any current loader — they need full per-layer blk.<il>.hc_attn_norm.weight and ship only blk.48 (verified 3×). Do not re-attempt without new loader support.
• The GPU budget on Dankclaw is the GTT aperture, not the rocm-smi "dedicated" figure. The Radeon 8060S (gfx1151) reports 116,732 MiB (llama-server --list-devices; gtt_total in /sys/class/drm/card1/device/), and the older "64 GiB, SVM-backed" lesson was already stale. free/dmidecode show the CPU-side carve-out — intentional, do not change. Dankclaw System Specs

Load / crash diagnosis

• Model load is slow (minutes cold) and health returns 503 while it runs: do NOT kill the process, deploy.sh waits up to 20 minutes (LLM_WAIT). On halogen the FIRST request after a (re)start can go silent for ~45–60 s (engine_unresponsive, 503s) and then self-recovers — do not restart over it. frank-dragoon
• Crash-loop procedure: (1) journalctl -u <svc> -n 100 AND the kernel line via journalctl -k (a segfault at address 8 in libamdhip64 is a NULL unref in HIP); (2) run the serve script, then the raw binary — a different outcome isolates the env file; (3) gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-<svc>.sh, because llama-server buffers stdout and loses the log tail on SIGSEGV: trust gdb, not the log; (4) amdgpu: SVM mapping failed spam is a symptom, not a root cause. LLM Performance Diagnosis
• Journal signatures: a client abort logs W srv stop: cancel task, id_task = N while a natural stop logs only slot release with truncated = 0. The server never preempts a running generation; excess tasks queue when slots are busy.

Host memory / swap

• Swap resize ordering: create and swapon the replacement BEFORE swapoff of the old one, or live pages must land in RAM that MemAvailable overstates. Dankclaw's 64 GiB /swap.img (fstab-driven) protects the co-tenants, not an endpoint with pinned weights — paging the KV pool is worse than the OOM it prevents. Host swap state is invisible to deploy.sh (no repo file): swapon --show before reasoning about headroom. Swap Configuration
• A sysfs raise is a countdown to the next reboot. ttm.pages_limit defaults to half of RAM, so a manual raise vanished with a power failure, frank-archon's pinned-ledger preflight refused the start, and Restart=always re-ran that refusal 6,919 times in ~20 h — a unit in activating (auto-restart) that exits 1 in 50 ms with no container and no engine log is a PREFLIGHT refusal, not an engine fault. Repo law now: scripts/ttm-preflight.sh raises the knob to FRANK_ARCHON_TTM_TARGET_MIB itself, re-reads, and refuses by name only after a write the kernel rejected. Never prescribe a manual sysfs write as the recovery procedure for a service that can boot before a human reaches it. frank-archon
• Silent RAM corruption outranks every software theory. frank-archon's long-conversation repetition/garbage was marginal DDR5 at 4800 MT/s on four DIMMs, fixed by dropping to 4600. Consumer non-ECC raises NO MCE for a silent flip, and a short in-band pattern check (60 s over 8 GiB of 246 GiB) is not a stability proof — both instruments said "ruled out" while the RAM was the cause, and "4800 is the JEDEC floor" is a rating, not a margin guarantee. When the fault is data-shaped (NaN states, garbage tokens) and a software revert does not move it, price in the RAM before the runtime; a revert tested under the same marginal hardware is itself confounded. frank-archon

Runtime-specific guards

• Rootless podman cannot RAISE rlimits, so --ulimit nofile=1048576 dies at container CREATE, not at runtime: runc reports error setting rlimit type 7: operation not permitted (type 7 = RLIMIT_NOFILE) when the requested hard limit exceeds the host process's ceiling (524288 on Dankclaw). Inherited-equal limits are fine, so memlock=-1:-1 works iff the unit grants LimitMEMLOCK=infinity — systemd is privileged and the value flows through podman to runc without any raise. Upstream halogen's own run lines pass NO --ulimit at all: copy vendor run lines before adding flags. frank-dragoon
• A GGUF's custom tensor type picks the runtime — read the histogram before promising a file to a loader. A non-standard tensor type in the KV section makes stock loaders refuse the file at load, so a fork can be the only door rather than a preference. Counting tensor types takes one python read and settles a runtime argument in minutes; build identity matters as much as the fork, so pin the commit by SHA in any build script. GGUF Compatibility
• Speculative-decode flags move as a SET. Handing the fork --model-draft with no --spec-type (the naive way to build a "no MTP" bench arm) segfaulted at load, core dumped, no diagnostic. A baseline arm must strip --model-draft AND every --spec-draft-* AND --spec-type, or it measures a crash instead of the absence of drafting. frank-zealot
• An orphaned scratch server taxes the whole host. A bench child left alive by a client-side exception kept ~26 GiB of weights resident on a 121 GiB box and pushed available to 3 GiB, so every later rate was suspect. Record the child pid, kill it in a finally, then prove free came back before believing a number — and kill by pid, never pkill -f llama-server.

llama.cpp serving

• MTP is a property of the FILE, not of the flags. The -mtp export carries the head as blk.<n>.nextn.* (qwen35.nextn_predict_layers = 1); --spec-type draft-mtp sets load_mtp and the draft context runs on the MAIN model's weights (shares_model), so there is no second copy of a 12 GiB model in VRAM and no -md argument. The plain build has no MTP tensors at all, and pointing --model-draft at that file loads it twice. frank-zealot
• What can be chained: ONE context-backed drafter plus any number of draftless ngram types. --spec-type a,b chains in a fixed internal order (CLI order irrelevant) and the gate is ctx_dft: the ngram family is added unconditionally while draft-mtp, draft-dflash, draft-dspark and draft-eagle3 all need a draft CONTEXT. So draft-dflash,ngram-mod is legal and it wins on prompt-echo traffic, but draft-mtp,draft-dflash dies at load — "context type MTP requested but model doesn't contain MTP layers". A LEGAL chain can still lose on a recurrent model: draft-mtp,ngram-mod on frank-dragoon's qwen35 hybrid loses, because adding the ngram layer logs recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support and MTP verification depends on that rollback. Measure chains per architecture, never by analogy. frank-dragoon
• A drafter's KV scales with the TARGET's context, and its window policy decides the price. The draft context is built at llama_n_ctx(ctx_tgt) with no flag to cap it, so a drafter declaring NO sliding window caches the whole window on --spec-draft-device and can OOM the first draft graph capture AFTER /health has answered. Read sliding_window out of the drafter's GGUF header before sizing, and prefer a skewed -ts over a context cut. Llama Cpp Serving Sizing
• A draft context bills its own f16 KV over the WHOLE POOL, not the slot, plus its own compute buffers, so on a single-card endpoint that squeeze can cost more than the drafter wins. The MTP question is never "does the file carry the head" but what draft KV + buffers cost against what the batch knobs leave for compute. Measure the arm, never the flag. frank-zealot
• A quant's scale overhead belongs in its byte rate; doubling a measured figure to "get the f16 number" invents 6.25 %. q8_0 is 1.0625 B/element (32 codes + one 2-byte scale per 32-block), f16 is exactly 2.0 ⇒ the ratio is 1.882×, not 2×. Derive byte rates from element count × block layout, then check them against the boot ledger — this repo carried an invented 2× number for weeks. Current KV math: frank-zealot
• --ubatch-size has an OPTIMUM, not a plateau, and the curve is BACKEND-SPECIFIC — never carry it across hosts. Its compute buffers land about TWICE as heavy on the card that holds the output layers and the drafter, so a split balanced at one batch is unbalanced at another: re-measure the balance after any batch change. The logical --batch-size is nearly free; a large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit, which no totals model can see. A prompt shorter than the ubatch cannot see the knob at all — that is the control that proves the knob, not the traffic, moved the number. The optimum is a property of the backend, so re-find it after any device or payload change. When a bench IS ordered, A/B it by restarting the SERVICE through a transient FRANK_<SVC>_ENV_FILE drop-in, never a scratch port: one card holds one server.
• One -ts step is one LAYER, and a hybrid stack's layers are not the same size (a DeltaNet boundary and a full-attention boundary can differ by ~1 GiB each with its own KV), so balance by measurement. An affine fit reproduces the deployed point and overstates the gaps in between. Llama Cpp Serving Sizing
• ngram-mod prints no startup banner. The load log names draft-dflash and nothing for the matcher, so a chain can look unengaged while it works. Identify it behaviourally (draft_n > 0 only where text repeats) or by the per-task draft acceptance = ... mean len = ... line. It costs no VRAM (a 64 MiB host-RAM hash table) and its window decides which traffic it can see: too short and it proposes on ordinary prose fragments, costing verification without acceptance. Llama Cpp Serving Sizing
• --parallel divides the pool, so MORE conversations cost WINDOW, and the per-slot number is the contract — not ctx-size. n_ctx_slot = CTX_SIZE / PARALLEL, and llama.cpp refuses a prompt longer than the SLOT with a 400 even when the pool has room, so models.yml contextWindow MUST follow every reslice. Both slot directions cost or buy window, never pool; shorter slots decode slightly faster. Measured reslices: frank-zealot.
• Price a VRAM budget from two boots, never from a formula — and an mmproj costs ~1.5× its file size (the encoder's activation and compute buffers ride with it). Subtracting two booted rows isolates it exactly; extrapolating weights+KV+"a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then solve KV ≤ card − non-KV − headroom and round to a multiple of PARALLEL × 4096. The projector is NOT billed in llama.cpp's own fit check: the engine prints its worst case and fits WITHOUT it, so price a shape from settled nvidia-smi after a real boot — the projector, and any drafter context, lands after /health would already have answered (same class as the DSpark drafter OOM). Two-boot table: frank-zealot.
• images and image_url are NOT interchangeable: only the content part reaches the model. --mmproj loads, usage.prompt_tokens rises, and the model still invents an answer if the client used the legacy top-level images field on /v1/chat/completions — frank-zealot answered "Brown." for a solid-red PNG that way and "Red." through the OpenAI image_url part. The failure is silent because a hallucinated color is a plausible answer; probe with a solid color and expect an exact word. frank-zealot
• --jinja is what tools needs. A subagent died with 500 tools param requires --jinja flag on the ik_llama.cpp runtime that served frank-dragoon: every tool-using agent on that endpoint was broken and the failure read like a model fault. Mainline llama.cpp defaults --jinja on and the serve scripts pin it; halogen also accepts tools and returns tool_calls. Check tool support before routing a role at an endpoint.
• A model's own template can REJECT the harness's thinking level. Qwen3.8's template accepts only xhigh, medium and low for reasoning_effort and raises on anything else — HTTP 500, not a clamp — so the harness's levels must be folded onto that set in models.yml (compat.reasoningEffortMap). A role whose level is unmapped fails on its FIRST request and the failure names Jinja, not the model: every subagent on frank-zealot died with Unexpected reasoning effort high until high: xhigh was added. Halogen accepts the wider set, so the two entries legitimately differ.
• Thinking tokens count against max_tokens, so a short-budget vision probe reads as blindness. A probe at max_tokens: 64 returned content: "" with finish_reason: "length" and half a thought in reasoning_content while the image HAD reached the model (322 prompt tokens vs 42 for the text alone). Budget ≥512 for any thinking model you probe for vision.
• The FIRST request after a boot is a warmup artifact, and the 3× spread on one shape makes graph-capture cost read as a model fault. Benchmark the second request or later. frank-zealot
• A fork build hides its own dtype laws. Vendor builds carry their own compile options, so prove a build's KV behaviour from the KV buffer line in the boot ledger and its CMakeCache.txt — never from the build flags you remember passing. frank-zealot
• A llama.cpp endpoint is invisible to Prometheus unless --metrics is passed: without it the engine answers 501 Not Implemented on /metrics, so the scrape fails while /health answers 200 and LlmEndpointDown fires beside a healthy service. halogen serves /metrics natively, so the asymmetry reads as "only the 3090 is broken". Diagnose with lastError from /api/v1/targets, not the alert list, and note Alertmanager has no persistence: a reboot re-stamps startsAt on an alert that has fired for weeks. Observability frank-zealot
• A leftover bench drop-in silently hijacks the next deploy, and no health gate sees it. The sanctioned bench-window method (a transient Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env drop-in sourcing the repo env) outlives the bench unless retired: a leftover frank-zealot.service.d/ptq1.conf made the cutover deploy boot a different model at a different shape while /health answered 200, /v1/models echoed the alias (a flag, not a fact) and the Playwright gate passed 33/33. After deploying any recently-benched LLM service: systemctl show <svc> -p DropInPaths (must be empty), then tr '\0' ' ' </proc/$(pidof llama-server)/cmdline, then the boot ledger — and retire drop-ins to ~/.AGENTTRASH/<date>-<topic>/ + daemon-reload. frank-zealot
• A bench verdict that never reaches the repo is indistinguishable from a rogue arm. That same arm served production for 24 minutes with real numbers (~48 t/s decode, 801 t/s prefill at 141k) that lived only in /tmp env files and journald, so the next agent read the surviving drop-in as an unfinished experiment and reported a hijack. Write the measurement into the endpoint page and the arm script's header in the session that takes it — /tmp and the journal are not records.
• llama-bench separates a LIST with / and SWEEP values with , — for both -dev and -ts. -dev CUDA0,Vulkan1 is two single-device configurations, never one two-device run, and -ts 0.85,0.15 is two sweep values each putting everything on device 0. Write -dev A/B -ts x/y. This artifact produced a wrong fleet conclusion ("CUDA + Vulkan cannot serve one model") from two bogus OOMs, and the bench table's own dev column was printing the truth the whole time. Any placement-dependent number needs an independent check: sample the other device's memory (nvidia-smi) while the run is loaded.

Ops

• Never pkill -f a pattern that appears in your own command line — it kills your ssh session. Use pkill -x <exact-name>.
• Both hosts allow passwordless sudo for the deploy user. Use plain sudo in committed scripts (no echo pw | sudo -S); FRANK_SUDO_PW covers hosts without it.
• ComfyUI on Dankclaw runs with GPU access (/dev/kfd + /dev/dri via AddDevice, GroupAdd=keep-groups, plus the udev rule that makes those nodes world-writable — user-unit spawns carry no supplementary groups) and must NOT pass --cpu: the gfx1151 ROCm wheels exist and a stale device-less container was found restarted 9,146 times. ComfyUI
• SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its ports off the gluetun service. SearXNG Gluetun
• Dankclaw's kernel command line is a repo file. It boots iommu=pt since 2026-10-18, tracked in dankclaw/host/grub, and moves ONLY through scripts/set-dankclaw-bootline.sh (diff, install, update-grub, verify grub.cfg, --reboot) — never by hand-editing /etc/default/grub. It replaced amd_iommu=off, and the local A/B on byte-identical traffic says identity mode costs nothing, so the XDNA2 NPU returned for free. dankclaw-iommu-translation-off
• Rootless Quadlet containers run with --replace --rm: podman logs covers only the CURRENT instance; history across recreations lives in journalctl --user -u <name>.service. Read rootless storage WITHOUT sudo — it is per-user storage, so sudo podman logs finds nothing. Quadlet Lifecycle
• A dead rootless podman pause process breaks ALL container management on the host, and podman system migrate panics in 5.7.0. Every podman command fails with invalid internal status, every start fails binding the bridges (failed to bind udp listener on 10.89.x.1:53), every Restart=always quadlet loops — while container storage stays intact. What is gone is /run/user/<uid>/libpod/tmp/alive (a UUID token) and the rootless netns; deleting the stale token is not enough. Repair: reboot (/run/user clears and the quadlets return on their own — check Linger=yes and WantedBy=default.target first). The 2026-09-16 occurrence had no OOM in journalctl -k: treat the pause process as host state that can die for its own reasons. Quadlet Lifecycle
• A container cannot reach a Network=host service at the host's LAN IP: netavark DNATs published ports only, so the host bind and the 10.89.1.1 gateway are both refused from a bridge container while the published port answers. Use host.containers.internal (169.254.1.2). This 502'd hermes.franksdank.cloud until its Caddyfile upstream changed. Container Host Reachability
• A *.gguf filter misses mradermacher's payloads: large quants ship as .partNofM fragments while the smaller rungs are single files, so filtering endswith(".gguf") makes a whole quant class look absent. Fetch the parts, verify each against the Hub's LFS sha256, concatenate in numeric order. Model Swap and Rollback
• A fetch script must -f and verify, or it reports success on an access error. curl -o FILE URL exits 0 on HTTP 401/403 and writes the error body as the "artifact": a gated repo yielded three 153-byte shards with identical sha256 and a cheerful DOWNLOAD_DONE in 20 seconds for 104 GiB. Use -f/--fail-with-body, compare each file's byte count to the Hub's declared size, and check the LFS sha256 — the live scripts/download-frank-*.sh scripts do all three; ad-hoc fetches tend not to. Gated repos 403 even with the host's HF token until that account accepts the gate. Model Swap and Rollback
• A verdict is a lead — the scout's and the tool's alike. In the 2026-10-19 documentation campaign six scouts worked the same pages and produced invented bullet IDs, files that do not exist, "duplications" grep disproved, line numbers past the end of the file they judged, and Eras tables with invented payloads. The mechanical tool was worse: scripts/port-census.py built its corpus from the directory containing the page under test, so each page was held by itself and ten era pages reported "0 sole-record lines" — which reads as "pure duplication, trash it". Nothing was cut on a verdict; every holder was re-grepped and every tool re-measured before it was believed.
• Never let a census decide what to keep. That same campaign then concluded the era pages were irreplaceable because each held 14–63 sole-record lines — wrong in the opposite direction: those were throughput figures for payloads that no longer run. Point of use is the only ownership test, the operator applies it, and 53 pages plus the 2,380-line change diary were destroyed as a result. A tool can say where a fact lives; it cannot say whether anything needs it.

Runbook Index

One page per live endpoint; every payload swap it has ever run is a row in that page's Eras table, with the rollback path in the same row. There is no per-attempt page and no per-swap ledger anywhere else.

Service	Page
frank-archon — radiance, 2× R9700 TP=2, Dankserver	serverAppConfigs-docs/LLM/frank-archon.md
frank-dragoon — halogen flash-server, Dankclaw iGPU	serverAppConfigs-docs/LLM/frank-dragoon.md
frank-zealot — llama.cpp (PrismML fork), RTX 3090	serverAppConfigs-docs/LLM/frank-zealot.md
frank-tts — cloned-voice speech, Dankclaw	serverAppConfigs-docs/Dankclaw/Frank TTS.md
Dankclaw kernel posture (iommu=pt since 2026-10-18)	serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md
Deployment flow	serverAppConfigs-docs/Architecture/Deployment Flow.md

There is no page for a retired endpoint and no change diary. A payload swap appends one row to the live endpoint's Eras table; when the endpoint itself dies, its page is destroyed. Rollback is a commit, not a document. serverAppConfigs-docs/index.md is the locator for every page that survives; retired scripts live in old/ and .AGENTTRASH/.

Why

Direct edits on the remote drift from version control, and future agents won't know what changed. Git is the single source of truth. The two workloads share two hosts, one Caddy, and one Gitea; one repo keeps them in one history.

Summary

Context	Action
Editing a compose/Caddyfile/serve script	Through git — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote.
Config exists only on remote	Pull it into the repo first — commit the fetched version (redacted) before editing.
Unstaged local changes	Commit or discard — check git status before starting work; ask if unsure.
Deploying a committed change	scripts/deploy.sh <target> — pushes, pulls ff-only, plans, dispatches, health-checks.
Container without a Quadlet unit	Give it one — boot-persistent, self-healing .container via git.
Changing a model/payload	Swap it — edit the trio, deploy, health + one chat/tool/vision probe. No benches unless Frank orders one; if it misbehaves, revert.
Writing a doc	One owner per fact — edit the endpoint/service page; everywhere else links. Never restate a number.
Benchmarking (only when ordered)	Stop and mask the service first, bench, unmask, restart, verify health.
Deleting a file	Move it to trash — .AGENTTRASH/ (in-repo) or ~/.AGENTTRASH/YYYYMMDD-topic. Never rm.
Tests exist	Run them — verify tests pass after every change. Do not commit failing tests.
Real secrets	Never commit — host .env files; .env.example holds placeholders only.