serverAppConfigs/AGENTS.md

AGENTS.md — serverAppConfigs Agent Rules

Overview

This repository is the single source of truth for the two homelab servers and the FrankBook-Pro Mac mini. It tracks the Podman container stacks and the LLM inference fleet. One repo, one checkout per host, one deploy script. All services run from /home/dank/serverAppConfigs on each host (the 2026-08-22 merge absorbed the former vllmScripts repo, history preserved).

• Dankserver (192.168.0.235) — media/library stack (RomM, Copyparty, Calibre-Web, Jellyfin, Plex) plus the frank-medium LLM service.
• Dankclaw (192.168.0.143) — AI/search/dev/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, Alloy, Pi-hole) plus TWO LLM services: frank-large (halogen-flash-server on the gfx1151 iGPU) and frank-small (llama.cpp on the RTX 3090). Caddy on Dankclaw reverse-proxies both hosts.
• FrankBook-Pro (192.168.0.112) — macOS mini running the *arr stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside its podman machine; all egress rides the Gluetun WireGuard netns. Deployed via the frankbook-pro handler.

Fleet topology (ACTIVE)

Target	Service	Runtime	Model	Port
Dankserver	frank-medium	vllm-radiance 0.9.3 (ggz14/radiance-vllm-mxfp4 patches), 2x single-GPU rootless Quadlets behind an adaptive-routing proxy	just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq + Qwen3.8-27B-DFlash2-FP8 drafter, 2x Radeon AI PRO R9700 (one card per replica, TP=1), 114688 ctx / 4 seqs each	7770
Dankclaw	frank-large	halogen-flash-server 0.5.8 (rootless Quadlet)	peonist-ai Qwen3.8-Flash-Next w4b .hgn on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only	7770
Dankclaw	frank-small	mainline llama.cpp master 37b53fd (root systemd unit)	ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (-mtp + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint	7771

Retired endpoints (frank-npu, and every prior runtime) are in the runbook index. The full swap history lives in serverAppConfigs-docs/LLM/index.md — do not grow it here.

Repository Layout

• dankserver/ — compose.yaml + Caddyfile + .env.example for the media stack
• dankclaw/ — compose.yaml + Caddyfile + .env.example for the AI stack
• shared/ — homepage, Prometheus, Loki, Alloy, SearXNG configs
• scripts/ — deploy.sh / deploy-plan.py / deploy-remote.sh, LLM serve scripts, systemd units, env tunables, bench harnesses
• old/ — retired LLM scripts (reference only)
• serverAppConfigs-docs/ — knowledge base (Obsidian vault, OKF format)
• tests/ — bats suite + Playwright e2e

Git Workflow

All edits flow through git. Never edit files on a remote machine directly.

1. Source the live config from the target machine
2. Commit the fetched version here (redact secrets into .env)
3. Edit locally
4. Test — ./tests/run.sh must pass
5. Commit and push — origin is Gitea on Dankclaw
6. Pull on the target (deploy.sh does this)
7. Deploy — scripts/deploy.sh plans and dispatches only changed services

Deployment

./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankserver
./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-medium
./scripts/deploy.sh dankclaw --dry-run

The planner maps changed paths to services; the remote runner restarts only the selected service and records its revision outside the checkout. Documentation-only changes are a no-op. LLM services are selected only by their own files — frank-large by dankclaw/frank-large/quadlet/, frank-small by its unit/env/serve trio in scripts/ — so unrelated container changes cannot interrupt inference. Set FRANK_SUDO_PW for headless runs on a host without passwordless sudo; the value never enters the repo.

Deployment completeness (MANDATORY)

A deploy is not complete when deploy.sh exits green. It requires:

1. ./tests/run.sh — all seams green, and
2. the Playwright e2e suite (tests/e2e/run.sh) — runs automatically as the [3/3] gate at the end of deploy.sh against the LIVE fleet; a behavior-affecting deploy that fails it exits DEPLOY INCOMPLETE. Bypass only with --skip-e2e in an emergency.

graph LR
    A[Edit locally] --> B[Tests pass] --> C[Commit + push]
    C --> D[deploy.sh plan] --> E[Target pulls ff-only]
    E --> F[Dispatch selected service] --> G[Per-service health]
    G --> H[Playwright e2e gate] --> I{green?}
    I -->|yes| J[Deploy complete]
    I -->|no| K[DEPLOY INCOMPLETE]

Boot Persistence & Self-Healing (MANDATORY)

Every deployed, working Podman container MUST have a self-healing, boot-persistent Quadlet unit. A container that runs only from compose.yaml or a manual podman run dies silently at reboot — rootless Podman does not auto-start bare containers, and restart: unless-stopped does not survive the explicit stop that shutdown performs. Never report such a service as "up".

Unit requirements:

• Boot persistence — [Install] WantedBy=default.target, installed to ~/.config/containers/systemd/<name>.container, enabled in the user manager, with linger on for the deploy user (loginctl show-user dank -p Linger → Linger=yes).
• Self-healing — Restart=always (or on-failure) with a short RestartSec; quadlet health check where the service has one.
• Repo provenance — the .container file lives in this repo (<host>/<svc>/quadlet/), is installed by the deployer, and is covered by the quadlet test seam. The host-side unit is generated state; the repo file is the source of truth.

Before declaring a service deployed: systemctl --user is-enabled <name> → enabled, is-active → active, and the container answers its health check. A running container with no unit is a gap: write the unit, test it, deploy it.

This rule exists because reboots found gaps twice (2026-08-26: Caddy as a bare compose container never returned; 2026-09-11: five observability services had no units). The observability services now have units under dankclaw/observability/quadlet/ and their compose definitions were removed — one owner per container name. The stack is configured as of 2026-09-16: a Grafana datasource and the llm-fleet dashboard are provisioned from shared/grafana/provisioning/, and shared/prometheus/rules/llm-fleet.yml loads 5 alert rules + 1 recording rule. Nothing is delivered — there is no Alertmanager, so a firing alert reaches nobody (an operator decision, not an oversight). Five Prometheus targets read down for structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is a blackbox exporter. See serverAppConfigs-docs/Infrastructure/Observability.md.

Testing

Tests are mandatory before commit (bats required on the dev machine):

./tests/run.sh

Seams: compose validity, env consistency (no leaked secrets), Caddyfile validity, quadlet units, docs completeness including vault link integrity, script syntax (shell + python), LLM unit integrity, LLM secret hygiene, deployment flow, service-scoped planning, homepage config/pages/deploy/checker, and e2e_scaffold — which fails if a Playwright spec file's number prefix is in no project, because such a spec silently never runs. See serverAppConfigs-docs/Architecture/Testing Policy.md.

Secret Rules

• Never commit real secrets. The live Gitea DB password and sudo password are test-enforced leak checks.
• .env files are git-ignored; commit only .env.example with placeholders. Exception: scripts/*.env LLM tunables (paths and numbers, never credentials).
• Do not document passwords anywhere in the repo. SSH keys are the auth mechanism; FRANK_SUDO_PW covers headless sudo and never enters the repo.

External Exposure Policy (MANDATORY)

Every Caddy-proxied service is either behind the auth chain (Anubis :8923 → Authentik embedded outpost) or explicitly internal-only — no public DNS name, no vhost. New services default to internal-only until guarded.

When guarding a new vhost — provider first, then route:

1. Create the outpost proxy provider FIRST (external_host = exact vhost URL with trailing slash; internal_host = http://caddy:8017). If the vhost flips first, authenticated traffic hits the outpost error page.
2. Flip the vhost to reverse_proxy anubis:8923 with header_up X-Real-Ip {remote_host}, and add the post-auth @<svc> host <vhost> handler on the http://:8017 listener routing to the real upstream.
3. Anubis guard set, Caddy TLS edge, and outpost provider list change together in one deploy.
4. Service-to-service traffic targets internal addresses (loki:3100), never the public hostname — non-browser clients cannot solve challenges.

Exempt categories (document the reason in the vhost comment): the IdP itself (auth.), git smart-HTTP endpoints, services with their own complete auth.

Deletion & Trash Policy (MANDATORY)

Never rm. Move deletions into trash so they stay recoverable:

1. In-repo files → .AGENTTRASH/ beside the item's directory.
2. Elsewhere → dated staging dir ~/.AGENTTRASH/YYYYMMDD-topic.
3. List staged items and sizes; wait for explicit user approval to purge.

rm, rm -rf, and git clean -f are forbidden without explicit permission. git * --no-verify is forbidden. The trash is a temporary hold, not an archive.

Benchmarking Policy (MANDATORY)

STOP the production service BEFORE benchmarking — these servers serve real users, and a bench beside a live service starves the GPU and corrupts results. Use the runtime's own harness: llama-bench (llama.cpp), vllm bench (vLLM), and halogen-flash-server's own bench/sweep modes for frank-large, which is the active runtime there again. Rolling your own bench requires asking first. Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself.

This covers any GPU experiment, not just benchmarks — including a bare 2-rank RCCL/torch.distributed probe. On 2026-09-16 one such probe was run against the live service "because it only needs 200 MB of VRAM"; the host went hard down ~20 minutes later (no ARP reply, 100% ICMP loss from two subnets, no recovery — a power cycle was required). A hung AQL dispatch wedging the GPU driver is the plausible mechanism, and this host has precedent for going unreachable until a power cycle. If an experiment needs GPUs, stop and mask first, or use a host that serves nobody. Telling an off machine from a wedged one: ip neigh show <ip> from a peer — FAILED means off/NIC dead, while a merely wedged kernel usually still answers ARP.

1. Stop and mask (self-healing would restart it):
   • Dankclaw frank-large (rootless Quadlet again since 2026-09-16): systemctl --user stop frank-large, then move its ~/.config/containers/systemd/frank-large.container aside and systemctl --user daemon-reload — the quadlet generator re-creates that unit from the file on every reload, so the source is the lever, not mask.
   • Dankclaw frank-small (root unit, llama.cpp on the 3090): sudo systemctl stop frank-small && sudo systemctl mask frank-small
   • Dankserver frank-medium (system unit): sudo systemctl stop frank-medium && sudo systemctl mask frank-medium
2. Bench.
3. Unmask and start; verify curl http://192.168.0.<host>:7770/health.

CRITICAL: never systemctl disable a system unit on these hosts — it deletes the unit file from /etc/systemd/system/, not just the symlink. mask symlinks to /dev/null and is safe/reversible.

Model Config (harness)

The harness reaches the fleet through .omp/agent/models.yml in the operator's home — /home/blank/.omp/agent/models.yml on the dev box, which is where the harness runs; there is no copy on the servers. Each provider is one HOST:PORT; each model id is the served alias: frank-medium → 192.168.0.235:7770, frank-large → 192.168.0.143:7770, frank-small → 192.168.0.143:7771. Per-role routing lives beside it in config.yml under modelRoles.

Resolution rules: the default model resolves as a defaultProvider + defaultModel pair (the provider must serve the model); a subagent override uses a bare model id, which resolves only when unique across providers. Two ports on one host need two providers — the file cannot express "same host, different port".

When renaming a service, retiring an endpoint, or moving a model: update the provider block, model id, contextWindow, maxTokens, input in models.yml, then fix every modelRoles entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As of 2026-09-16 the image path is frank-small (input: [text, image]), and frank-large is text-only because its halogen deployment loads no vision tower. There is no audio input anywhere in the fleet: that capability left with the FLM NPU endpoint on 2026-09-11.

Active Services

frank-large — halogen-flash-server (Dankclaw, ACTIVE)

halogen-flash-server 0.5.8 as a rootless Quadlet again since 2026-09-16 (dankclaw/frank-large/quadlet/frank-large.container), serving peonist-ai/halogen-qwen3.8-flash-next (115.55 GiB w4b .hgn checkpoint + the 2.31 GiB quality overlay) on port 7770 from the gfx1151 iGPU. Two conversations of 200,192 tokens resident (pool 400,384, 2 slots), greedy default, reasoning extracted to reasoning_content, and no vision tower — that moved to frank-small. It replaced ik_llama.cpp, which had replaced it for two days.

Measured boot ledger (2026-09-16, HALOGEN_VERBOSE=1 ships in the unit): device budget 121 − 67.7 − 20 = 33.5 GiB against a need of ~27.2 GiB, so the configured pool was not clamped; 68.0 GiB of weights locked + 11.0 GiB of KV pool + 16.5 GiB of working memory = 95.4 GiB in all, leaving 14.7 GiB (475 contiguous 2 MiB blocks) for everything else: frank-small on the 3090 and ~28 co-tenant containers. The engine printed its co-tenant stall NOTE and logged 4,350 compaction stalls (285 failed) while reserving the pool, so read that as "tight and fragmented", not "out of memory".

Runbook: serverAppConfigs-docs/LLM/halogen-flash-server.md. Rollback: frank-large-ik-uncensored.

frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)

Mainline llama.cpp master 37b53fd as a root systemd unit (repo scripts/frank-small.service, relink_united into /etc/systemd/system/), serving ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S — the -mtp build, so the model's own Multi-Token Prediction head drives speculative decoding — with the BF16 mmproj, on port 7771, 131,072 ctx / 1 slot, f16 KV, greedy default. Deployed flags are in scripts/frank-small.env; the serve script refuses to start if the weights, the projector or the binary are missing.

Verified at deploy (2026-09-16): /v1/models reports frank-small; a chat completion returns content plus reasoning_content; a tool call returns tool_calls; the slot timing lines show draft acceptance = 0.66–0.94, mean len ≈ 3 (MTP engaged); an image request is answered from the projector while the halogen side refuses images with a named reason; 22,362 MiB of the card's 24,576 MiB is in use; n_slots = 1, n_ctx_slot = 131072.

Runbook: serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md.

frank-medium — vllm-radiance MXFP4 (Dankserver, ACTIVE)

stilldeadcode/vllm-radiance:0.9.3 plus the ggz14/radiance-vllm-mxfp4 patches (pinned clone at /home/dank/GitPackages/radiance-vllm-mxfp4; its patches are applied at every container start), serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with the Qwen3.8-27B-DFlash2-FP8 block-diffusion drafter (SPEC=7).

The deployment is two single-GPU replicas behind an adaptive-routing proxy — frank-medium-a (:7771, R9700 #1), frank-medium-b (:7772, R9700 #2) and frank-medium-proxy (:7770, the fleet endpoint), three rootless Quadlets under dankserver/frank-medium/quadlet/. Each replica is TP=1 at 114688 context / 4 sequences with vLLM-profiled KV (KV_MEM=0).

TP=2 across both cards cannot run on this host. RCCL's collective kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to dispatch them on a device without PCIe atomics; the second R9700 sits behind the chipset bridges 05:00.0/06:09.0, which advertise AtomicOpsCap: Routing- — a read-only capability, so no software can enable atomics on that path. Not the kernel version (7.0.0-28/-30/-31 identical), not the image, not the RCCL build (a from-source rebuild fails identically). Putting both cards on CPU root ports restores TP=2: on this board that means an M.2→PCIe x4 riser into the free, CPU-attached M.2_2 slot, or a board whose CPU x16 bifurcates to x8/x8. frank-medium.container stays in the repo as the TP=2 target; the deploy handler does not install it.

Runbook: serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md. Prior runtime: frank-medium-rocmfpx (llama.cpp era, same day).

Key Lessons Learned

Durable operational lessons from the fleet's history. Full detail in the linked runbooks.

PCIe / multi-GPU

• RCCL collectives need PCIe atomics on every participating GPU. RCCL's kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch them when the device has no atomics, and the failure surfaces as hipErrorIllegalState ("the operation cannot be performed in the present state") from hipLaunchKernel, not as anything that names atomics. The value is consumed at driver probe time, so writing the AtomicOp Requester bit with setpci at runtime changes nothing. Check the chain: lspci -vv -s <gpu> and every upstream bridge must show AtomicOpsCap: Routing+ (a Routing- bridge is a hardware wall), and the endpoint's Device Control 2 (setpci -s <gpu> CAP_EXP+28.w) must have bit 6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the chipset (usually not) — that asymmetry is what blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL short-circuits at world_size=1.
• The fix is physical — and cheap on this board. Two paths put both cards on CPU root ports, which restores the atomics routing and therefore TP=2: an M.2→PCIe x4 riser into a CPU-attached M.2 slot (on dankserver's TUF GAMING B650-E WIFI that is M.2_2, which is free — per ASUS's spec page; PCIEX16_2, where card #2 sits today, is chipset and x1-mode), or a board whose CPU x16 bifurcates to x8/x8. Bandwidth is not the constraint: a 27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4. RCCL's P2P send/recv fails on such a card too, so pipeline parallelism is not an escape either. Full evidence matrix, the open rebuild hypothesis, and the test plan: Dual-R9700-Cooperation.
• When a GPU collective dies, read the runtime, not the framework. The message that identified the cause came from AMD_LOG_LEVEL=3 (Pcie atomics not enabled, hostcall not supported → AQL dispatch failed!), not from vLLM or RCCL. AMD_LOG_LEVEL costs one env var and prints the failing HIP call with its return code.

ROCm / GPU

• Layer split, never tensor, on the RCCL-less ROCmFPX builds. Tensor parallel falls back to a butterfly allreduce and collapses prefill ~7x (measured 271.8 vs 1991 t/s). Pin the MTP draft head to ROCm1 — an unpinned draft under tensor mode aborts the KV build.
• mmap must stay on for the big-PLE models: --no-mmap forces the ~50 GiB PLE table into anonymous memory → silent cgroup kill, dmesg is the only evidence. mmap pages ARE page cache. Never pin the PLE tensor with -ot (23 → 13.4 t/s).
• HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE. Serve scripts export env files with set -a, so a tunable named TEMP leaks into HIP (TEMP=1.0 sent the JIT cache to path 1.0 and segfaulted every load, fixed 2026-08-13 by renaming to TEMPERATURE). Never name a tunable these; check with strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE".
• -DGGML_CUDA_FA_ALL_QUANTS=ON is required for q8_0 KV + flash attention on CUDA builds; without it prefill collapses ~12x (95 vs 1300+ t/s, measured 2026-08-17).
• Flash-Next MTP heads do not load on any current loader — they need full per-layer blk.<il>.hc_attn_norm.weight and ship only blk.48 (verified 3x). Don't re-attempt without new loader support.
• QSA flash-attn assert: GGML_ASSERT(a-type == GGML_TYPE_F32) on qwen4exp needs scripts/patches/qwen4exp-qsa-f32-mask.patch — but only on the agentionai FAST layer set; the kingjones STRIX set works unpatched.
• GPU budget on Dankclaw is the rocm-smi figure (64 GiB, SVM-backed). free/dmidecode show the CPU-side carve-out — intentional, do not change.

Load / crash diagnosis

• Model load is slow (minutes cold); health returns 503 while loading; do not kill the process. deploy.sh waits up to 20 minutes (LLM_WAIT). On halogen, the FIRST request after a (re)start can go silent ~45–60 s (health engine_unresponsive, 503s) and then self-recovers — do not restart over it.
• Crash-loop procedure: (1) journalctl -u <svc> -n 100 + kernel line via journalctl -k (segfault at address 8 in libamdhip64 = NULL unref in HIP); (2) run serve script, then raw binary — different outcome isolates the env file; (3) gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-<svc>.sh — llama-server buffers stdout and loses the log tail on SIGSEGV; trust gdb, not the log; (4) amdgpu: SVM mapping failed spam is a symptom, not a root cause.
• Journal signatures: client abort logs W srv stop: cancel task, id_task = N; natural stop logs only slot release with truncated = 0. The server never preempts a running generation; excess tasks queue when slots are busy.

Host memory / swap

• Swap resize ordering: create and swapon the replacement BEFORE swapoff of the old one, or live pages must land in RAM that MemAvailable overstates. Dankclaw swap (64 GiB at /swap.img, fstab-driven) protects the co-tenants, not frank-large — pinned weights don't swap, and paging the KV pool is worse than the OOM it prevents.
• Host swap state is invisible to deploy.sh (no repo file) — check swapon --show before reasoning about headroom.

Runtime-specific guards

• Kairic Edge (if ever re-deployed): the pinned CK commit needs third_party/composable_kernel/include/ck/config.h generated manually (untracked; git clean removes it), and its greedy-argmax fastpath rejects non-greedy requests and reasoning budgets. Re-test chat before production.
• FastMTP sidecar: build must be commit 4df29be4 + the HauhauCS patch; stock binaries fail the draft load with a tensor-shape mismatch.

llama.cpp serving

• MTP is a property of the FILE, not of the flags. The -mtp GGUF build carries the head as blk.<n>.nextn.* (qwen35.nextn_predict_layers = 1); --spec-type draft-mtp then sets load_mtp, and the draft context runs on the MAIN model's weights (shares_model), so there is no second copy of a 12 GiB model in VRAM and no -md argument. The plain build has no MTP tensors at all, and pointing --model-draft at the file would load it twice.
• --jinja is what tools needs. On 2026-09-16 a subagent died with 500 tools param requires --jinja flag, from the ik_llama.cpp runtime that then served frank-large: every tool-using agent on that endpoint was broken and the failure read like a model fault. Mainline llama.cpp 0.4.1 defaults --jinja on and the serve script pins it; halogen also accepts tools and returns tool_calls. Check tool support before routing a role at an endpoint.
• Quantized KV on CUDA needs GGML_CUDA_FA_ALL_QUANTS=ON — the same option and the same ~12x prefill collapse described in the ROCm note above. f16 KV needs nothing, which is why frank-small runs f16.
• A model's own template can REJECT the harness's thinking level. Qwen3.8's template accepts only xhigh, medium and low for reasoning_effort and raises on anything else — HTTP 500, not a clamp — so the harness's levels must be folded onto that set in models.yml (compat.reasoningEffortMap). A role whose level is unmapped fails on its FIRST request, and the failure names Jinja, not the model: every subagent on frank-small died with Unexpected reasoning effort high until high: xhigh was added on 2026-09-16. Halogen accepts the wider set, so the two entries legitimately differ.

Ops

• Never pkill -f a pattern that appears in your own command line — it kills your ssh session. Use pkill -x <exact-name>.
• Both hosts allow passwordless sudo for the deploy user. Use plain sudo in committed scripts (no echo pw | sudo -S); FRANK_SUDO_PW covers hosts without it.
• ComfyUI on Dankclaw runs with GPU access (/dev/kfd + /dev/dri via AddDevice, GroupAdd=keep-groups, plus the udev rule that makes those nodes world-writable — user-unit spawns carry no supplementary groups). It must NOT pass --cpu: the gfx1151 ROCm wheels exist and the unit's --cpu era ended 2026-09-16, when a stale device-less container was found restarted 9,146 times (RuntimeError: No HIP GPUs are available).
• SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its ports off the gluetun service.
• amd_iommu=off on Dankclaw (bought 13–16 % prefill for frank-large, 2026-09-11) permanently retired the XDNA2 NPU: the driver's SVA/PASID bind needs an enabled IOMMU (/dev/accel/accel0 fails err=-19). Revert path in serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md.
• Rootless Quadlet containers run with --replace --rm: podman logs covers only the CURRENT instance; history across recreations lives in journalctl --user -u <name>.service. Read rootless storage WITHOUT sudo (per-user storage — sudo podman logs finds nothing).

Runbook Index

Active:

Service	Runbook
frank-large (halogen-flash-server, Dankclaw)	serverAppConfigs-docs/LLM/halogen-flash-server.md
frank-small (llama.cpp Qwen3.8-27B GSQ-RCO, Dankclaw)	serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md
frank-medium (vllm-radiance MXFP4, Dankserver)	serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md
Dankclaw kernel posture (amd_iommu=off)	serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md
Deployment flow + incidents	serverAppConfigs-docs/LLM/deployment.md

Retired (rollback targets and history):

Runtime	Host	Window	Runbook
kingjones STRIX Qwen3.8-Flash-Next ROCmFP4	Dankclaw	2026-08-30 → 09-11	kingjones-strix-rocmfp4.md
ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (frank-large)	Dankclaw	2026-09-15 → 09-16	frank-large-ik-uncensored.md
agentionai ROCmFP4-FAST imatrix	Dankclaw	2026-08-29 → 08-30	agentionai-fast-imatrix.md
eemin Qwen3.8-27B ROCmI4 W4A4 MTP	Dankclaw	2026-08-18 → 08-25	eemin-rocmi4-mtp.md
Kairic Edge IU4	Dankclaw	2026-08-22 (same-day revert)	kairic-edge.md
CIRU ActiveFPX v3 / PromptForge	Dankclaw	through 2026-08-22	ciru-activefpx-promptforge.md
DFlash2 Vulkan	Dankclaw	2026-08-21 (same-day revert)	dflash2-large-vulkan.md
FastFlowLM NPU (frank-npu)	Dankclaw	twice, retired 2026-09-11	frank-npu.md, FastFlowLM NPU Inference.md
FreeToken (official + Huihui FP8, frank-small)	Dankserver	2026-09-02 → 09-15	freetoken-frank-small.md
FastMTP patched llama.cpp	Dankserver	retired 2026-08-18	qwen3.8-spec-matrix.md
ik_llama.cpp Ornith CRACK	Dankserver	retired 2026-09-02	ikllama-ornith-dankserver.md

Retired runbooks are under serverAppConfigs-docs/LLM/. Retired scripts live in old/ and .AGENTTRASH/.

Why

Direct edits on the remote drift from version control, and future agents won't know what changed. Git is the single source of truth. The two workloads share two hosts, one Caddy, and one Gitea; one repo keeps them in one history.

Summary

Context	Action
Editing a compose/Caddyfile/serve script	Through git — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote.
Config exists only on remote	Pull it into the repo first — commit the fetched version (redacted) before editing.
Unstaged local changes	Commit or discard — check git status before starting work; ask if unsure.
Deploying a committed change	scripts/deploy.sh <target> — pushes, pulls ff-only, plans, dispatches, health-checks.
Container without a Quadlet unit	Give it one — boot-persistent, self-healing .container via git.
Benchmarking	Stop and mask the service first, bench, unmask, restart, verify health.
Deleting a file	Move it to trash — .AGENTTRASH/ (in-repo) or ~/.AGENTTRASH/YYYYMMDD-topic. Never rm.
Tests exist	Run them — verify tests pass after every change. Do not commit failing tests.
Real secrets	Never commit — host .env files; .env.example holds placeholders only.