serverAppConfigs/AGENTS.md

AGENTS.md — serverAppConfigs Agent Rules

Overview

This repository is the single source of truth for the two homelab servers and the FrankBook-Pro Mac mini. It tracks the Podman container stacks and the LLM inference fleet. One repo, one checkout per host, one deploy script. All services run from /home/dank/serverAppConfigs on each host (the 2026-08-22 merge absorbed the former vllmScripts repo, history preserved).

• Dankserver (192.168.0.235) — media/library stack (RomM, Copyparty, Calibre-Web, Jellyfin, Plex) plus the frank-primary LLM service.
• Dankclaw (192.168.0.143) — AI/search/dev/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, Alloy, Pi-hole) plus TWO LLM services: frank-large (stock llama.cpp on the gfx1151 iGPU) and frank-small (SGLang on the RTX 3090). Caddy on Dankclaw reverse-proxies both hosts.
• FrankBook-Pro (192.168.0.112) — macOS mini running the *arr stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside its podman machine; all egress rides the Gluetun WireGuard netns. Deployed via the frankbook-pro handler.

Fleet topology (ACTIVE)

Target	Service	Runtime	Model	Port
Dankserver	frank-primary	tcclaviger RDNA4 vLLM fork (docker.io/tcclaviger/vllm@sha256:da4029666…, digest-pinned, plain podman run, root systemd unit)	orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, fp8 KV (--kv-cache-dtype fp8), no speculative decode, MNBT 2048, MoE gather 32×64, 3 seqs × 262144 ctx (the model's full native window) at KV_BYTES 6,100,000,000/rank — the pool pinned to the contract FLOOR so everything else feeds the expert arena — util 0.98, and the client contract wired: --reasoning-parser qwen3 (thinking in reasoning, never inline), --enable-auto-tool-choice --tool-call-parser qwen3_xml, default chat-template kwargs enable_thinking + preserve_thinking. Five full-length contexts BOOT but leave ~0 bytes free at runtime — a real client request (995-token prompt, max_tokens=261149) killed the worker with torch.OutOfMemoryError ... 0 bytes free; six never fit at all (6×241664 asked 10.70 GiB vs 9.18 GiB free). Three slots with a floor-sized pool is the deployed shape. DCP cannot help here: it requires TP > the model's 2 KV heads, i.e. four GPUs. fp8+MTP4 shape restorable from git tag frank-primary-fp8-mtp-20261001	7770
Dankclaw	frank-large	gufo gufo-org/gufo v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4)	Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only	7770
Dankclaw	frank-small	SGLang in a self-contained python3.12 venv (root systemd unit; the venv carries its own CUDA 13 stack — see the runbook)	letechlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, 12.0 GiB, INT8 W8A16 AutoRound, in-checkpoint vision tower, NO MTP weights shipped so no spec decode) on the RTX 3090, fp8 KV, pinned 606,208-token pool / 4 slots of 151,552, bf16 mamba states, eager prefill, 83.9 tok/s decode	7771

Retired endpoints (frank-npu, and every prior runtime) are in the runbook index. The full swap history lives in serverAppConfigs-docs/LLM/index.md — do not grow it here.

Repository Layout

• dankserver/ — compose.yaml + Caddyfile + .env.example for the media stack
• dankclaw/ — compose.yaml + Caddyfile + .env.example for the AI stack
• shared/ — homepage, Prometheus, Loki, Alloy, SearXNG configs
• scripts/ — deploy.sh / deploy-plan.py / deploy-remote.sh, LLM serve scripts, systemd units, env tunables, bench harnesses
• old/ — retired LLM scripts (reference only)
• serverAppConfigs-docs/ — knowledge base (Obsidian vault, OKF format)
• tests/ — bats suite + Playwright e2e

Git Workflow

All edits flow through git. Never edit files on a remote machine directly.

1. Source the live config from the target machine
2. Commit the fetched version here (redact secrets into .env)
3. Edit locally
4. Test — ./tests/run.sh must pass
5. Commit and push — origin is Gitea on Dankclaw
6. Pull on the target (deploy.sh does this)
7. Deploy — scripts/deploy.sh plans and dispatches only changed services

Deployment

./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankserver
./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-primary
./scripts/deploy.sh dankclaw --dry-run

The planner maps changed paths to services; the remote runner restarts only the selected service and records its revision outside the checkout. Documentation-only changes are a no-op. LLM services are selected only by their own files — every endpoint (frank-large/frank-small/frank-primary) by its unit/env/serve trio in scripts/ — so unrelated container changes cannot interrupt inference. Set FRANK_SUDO_PW for headless runs on a host without passwordless sudo; the value never enters the repo.

Deployment completeness (MANDATORY)

A deploy is not complete when deploy.sh exits green. It requires:

1. ./tests/run.sh — all seams green, and
2. the Playwright e2e suite (tests/e2e/run.sh) — runs automatically as the [3/3] gate at the end of deploy.sh against the LIVE fleet; a behavior-affecting deploy that fails it exits DEPLOY INCOMPLETE. Bypass only with --skip-e2e in an emergency.

graph LR
    A[Edit locally] --> B[Tests pass] --> C[Commit + push]
    C --> D[deploy.sh plan] --> E[Target pulls ff-only]
    E --> F[Dispatch selected service] --> G[Per-service health]
    G --> H[Playwright e2e gate] --> I{green?}
    I -->|yes| J[Deploy complete]
    I -->|no| K[DEPLOY INCOMPLETE]

Boot Persistence & Self-Healing (MANDATORY)

Every deployed, working Podman container MUST have a self-healing, boot-persistent Quadlet unit. A container that runs only from compose.yaml or a manual podman run dies silently at reboot — rootless Podman does not auto-start bare containers, and restart: unless-stopped does not survive the explicit stop that shutdown performs. Never report such a service as "up".

Unit requirements:

• Boot persistence — [Install] WantedBy=default.target, installed to ~/.config/containers/systemd/<name>.container, enabled in the user manager, with linger on for the deploy user (loginctl show-user dank -p Linger → Linger=yes).
• Self-healing — Restart=always (or on-failure) with a short RestartSec; quadlet health check where the service has one.
• Repo provenance — the .container file lives in this repo (<host>/<svc>/quadlet/), is installed by the deployer, and is covered by the quadlet test seam. The host-side unit is generated state; the repo file is the source of truth.

Before declaring a service deployed: systemctl --user is-enabled <name> → enabled, is-active → active, and the container answers its health check. A running container with no unit is a gap: write the unit, test it, deploy it.

This rule exists because reboots found gaps twice (2026-08-26: Caddy as a bare compose container never returned; 2026-09-11: five observability services had no units). The observability services now have units under dankclaw/observability/quadlet/ and their compose definitions were removed — one owner per container name. The stack is configured and delivering as of 2026-09-17: a Grafana datasource and the llm-fleet dashboard are provisioned from shared/grafana/provisioning/, shared/prometheus/rules/ llm-fleet.yml loads 5 alert rules + 1 recording rule, and firing rules are delivered: Prometheus → Alertmanager (dankclaw/observability/quadlet/ alertmanager.container, loopback :9093) → host-side Signal relay (scripts/signal-relay.py, user unit, 192.168.0.143:18093 — the host LAN IP, the one bind a container can reach as host.containers.internal; iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311. Prometheus targets read down for structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is a blackbox exporter. See serverAppConfigs-docs/Infrastructure/Observability.md.

Testing

Tests are mandatory before commit (bats required on the dev machine):

./tests/run.sh

Seams: compose validity, env consistency (no leaked secrets), Caddyfile validity, quadlet units, docs completeness including vault link integrity, script syntax (shell + python), LLM unit integrity, LLM secret hygiene, deployment flow, service-scoped planning, homepage config/pages/deploy/checker, and e2e_scaffold — which fails if a Playwright spec file's number prefix is in no project, because such a spec silently never runs. See serverAppConfigs-docs/Architecture/Testing Policy.md.

Secret Rules

• Never commit real secrets. The live Gitea DB password and sudo password are test-enforced leak checks.
• .env files are git-ignored; commit only .env.example with placeholders. Exception: scripts/*.env LLM tunables (paths and numbers, never credentials).
• Do not document passwords anywhere in the repo. SSH keys are the auth mechanism; FRANK_SUDO_PW covers headless sudo and never enters the repo.
• A token passed as curl -H "Authorization: Bearer …" is world-readable. It lands in /proc/<pid>/cmdline, so any ps/pgrep -a on the host prints it — that leaked an HF token into a session transcript on 2026-09-16, while merely checking whether a download was still running. Pass secrets out of band (curl -K <mode-600 file> carrying the header, or HF_TOKEN in the environment for hf/hf_transfer) and never pgrep -a a fetch command that carries an auth header.

External Exposure Policy (MANDATORY)

Every Caddy-proxied service is either behind the auth chain (Anubis :8923 → Authentik embedded outpost) or explicitly internal-only — no public DNS name, no vhost. New services default to internal-only until guarded.

When guarding a new vhost — provider first, then route:

1. Create the outpost proxy provider FIRST (external_host = exact vhost URL with trailing slash; internal_host = http://caddy:8017). If the vhost flips first, authenticated traffic hits the outpost error page.
2. Flip the vhost to reverse_proxy anubis:8923 with header_up X-Real-Ip {remote_host}, and add the post-auth @<svc> host <vhost> handler on the http://:8017 listener routing to the real upstream.
3. Anubis guard set, Caddy TLS edge, and outpost provider list change together in one deploy.
4. Service-to-service traffic targets internal addresses (loki:3100), never the public hostname — non-browser clients cannot solve challenges.

Exempt categories (document the reason in the vhost comment): the IdP itself (auth.), git smart-HTTP endpoints, services with their own complete auth.

Deletion & Trash Policy (MANDATORY)

Never rm. Move deletions into trash so they stay recoverable:

1. In-repo files → .AGENTTRASH/ beside the item's directory.
2. Elsewhere → dated staging dir ~/.AGENTTRASH/YYYYMMDD-topic.
3. List staged items and sizes; wait for explicit user approval to purge.

rm, rm -rf, and git clean -f are forbidden without explicit permission. git * --no-verify is forbidden. The trash is a temporary hold, not an archive.

Benchmarking Policy (MANDATORY)

STOP the production service BEFORE benchmarking — these servers serve real users, and a bench beside a live service starves the GPU and corrupts results. Use the runtime's own harness: llama-bench for the llama.cpp endpoints, gufo bench for frank-large. Rolling your own bench requires asking first. Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself.

gufo bench cannot rank speculative-decode settings, and will invert the ranking. Measured 2026-09-28 on frank-large at depth 32768: its corpus reports acceptance=0.933 ± 0.000 with an identical trace sha256 across draft widths, i.e. the same highly-predictable text every run, and 93 % of every proposal accepted. It reported 29–37 t/s decode where real traffic gives ~15 t/s, and it made --draft-tokens 16 look 6 % faster than 7 when real text made it 14 % slower. Use it for shape, not for A/B. For an A/B, replay real text through the live server and read the Server-Timing header — but note the unit rides in the name half (ttft;dur=402.485), so splitting on = alone keys the dict ttft;dur and every lookup silently returns nothing. Generate at least 128 tokens for the measurement: at 8 tokens, inter_token is first-token noise and the run-to-run spread swamps every real difference.

This covers any GPU experiment, not just benchmarks — including a bare 2-rank RCCL/torch.distributed probe. On 2026-09-16 one such probe was run against the live service "because it only needs 200 MB of VRAM"; the host went hard down ~20 minutes later (no ARP reply, 100% ICMP loss from two subnets, no recovery — a power cycle was required). A hung AQL dispatch wedging the GPU driver is the plausible mechanism, and this host has precedent for going unreachable until a power cycle. If an experiment needs GPUs, stop and mask first, or use a host that serves nobody. Telling an off machine from a wedged one: ip neigh show <ip> from a peer — FAILED means off/NIC dead, while a merely wedged kernel usually still answers ARP.

1. Stop and mask (self-healing would restart it):
   • Dankclaw frank-large (root unit, gufo on the iGPU): sudo systemctl stop frank-large && sudo systemctl mask frank-large
   • Dankclaw frank-small (root unit, SGLang on the 3090): sudo systemctl stop frank-small && sudo systemctl mask frank-small
   • Dankserver frank-primary (root unit, tcclaviger vLLM on the 2x R9700): sudo systemctl stop frank-primary && sudo systemctl mask frank-primary
2. Bench. NOTE: systemctl mask FAILS on a relinked unit (the unit file in /etc/systemd/system/ is itself a symlink into the checkout — mask refuses to replace it). Stop is enough for the bench window; the deploy restores the unit, so re-deploy after benching instead of unmasking.
3. Unmask and start; verify curl http://192.168.0.<host>:7770/health.

CRITICAL: never systemctl disable a system unit on these hosts — it deletes the unit file from /etc/systemd/system/, not just the symlink. mask symlinks to /dev/null and is safe/reversible.

Model Config (harness)

The harness reaches the fleet through .omp/agent/models.yml in the operator's home — /home/blank/.omp/agent/models.yml on the dev box, which is where the harness runs; there is no copy on the servers. Each provider is one HOST:PORT; each model id is the served alias: frank-primary → 192.168.0.235:7770, frank-large → 192.168.0.143:7770, frank-small → modelRoles.

Resolution rules: the default model resolves as a defaultProvider + defaultModel pair (the provider must serve the model); a subagent override uses a bare model id, which resolves only when unique across providers. Two ports on one host need two providers — the file cannot express "same host, different port".

When renaming a service, retiring an endpoint, or moving a model: update the provider block, model id, contextWindow, maxTokens, input in models.yml, then fix every modelRoles entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As of 2026-10-01 both frank-small and frank-primary accept images (input: [text, image]) — frank-primary loads the BF16 mmproj explicitly (this build never autoloads projectors), and frank-small's MiMo payload bakes its vision tower INTO the checkpoint (no projector file at all); frank-large remains text-only because its gufo deployment loads no vision projector. Note also that the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3 parser, live-verified) emit reasoning_content, vLLM emits reasoning — models.yml sets reasoningContentField per endpoint to match. There is no audio input anywhere in the fleet: that capability left with the FLM NPU endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's audio encoder, and left again with the Gemma4 retirement on 2026-10-01 (the MiMo tower is vision-only).

frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (Dankclaw, ACTIVE)

gufo (gufo-org/gufo) v0.1.1 — "the Strix Halo inference engine", a vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151) — as a root systemd unit (scripts/frank-large.service, User=dank), serving ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF's UD-Q8_K_XL (27B dense, 29.3 GiB, sha256-verified; abliterated/uncensored, and under the non-OSI SWIFT OPEN LICENSE 1.0) on port 7770 from the gfx1151 iGPU. One session, full 262,144 ctx (the model's native window, SESSIONS=1), DFlash2 spec decode (the z-lab Qwen3.8-27B-DFlash2-Q4_K_M drafter, gufo's adaptive block policy, --draft-tokens 7), greedy sampling, reasoning extracted to reasoning_content, and no vision projector.

Two swaps landed on 2026-09-28, both same day: the engine (from the unsloth llama.cpp b11030 prebuilt, which had replaced Kairic Edge ROCmFPX) and then the payload (from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL, still on disk and the payload-only rollback). The two targets are structurally interchangeable for gufo — same qwen35 architecture, the same 866 tensors in the same order, the same per-tensor Q8_0/F32/BF16 layout, the same MTP head at blk.64 — verified by parsing and diffing both GGUF headers before either was loaded. The DFlash2 drafter did not move, so it has its own DRAFTER_DIR and the deployer's stray-run sweep keys on the drafter's filename, not the target's.

The runtime is built from source on the host — /home/dank/GitPackages/gufo-v0.1.1/, reproducible with scripts/build-gufo.sh (--verify re-checks an installed tree). gufo ships no prebuilt Linux binaries and the v0.1.0/v0.1.1 release assets are empty. Unlike the retired llama.cpp prebuilt, which bundled its own ROCm, gufo links the host ROCm 7.2.4 through its RPATH, so the HIP dev packages must stay installed. Two host workarounds the build script encodes:

• Ubuntu's librocwmma-dev must be absent. It installs a truncated rocWMMA header set into /usr/include that shadows AMD's complete one and breaks the build with 'internal/accessors.hpp' file not found. A -I cannot fix it — in HIP mode the host pass searches the system directories first. The package has no reverse dependencies.
• --gcc-install-dir must name GCC 15. Ubuntu 26.04 ships /usr/include/c++/16, which the ROCm clang prefers; its <atomic_wait.h> needs a std::__detail::__notify_impl symbol version the installed libstdc++.so.6 (GLIBCXX_3.4.35) does not export, and the link fails.

gufo diagnose is the preflight and must report [PASS] with gfx1151. It also self-caps compute queues (GPU_MAX_HW_QUEUES, 2 resident for serve llm) after reading /sys/class/kfd/kfd/proc/*/queues/*/type, which matters because ComfyUI holds queues on the same device; four HIP processes is the device ceiling.

The client contract changed, and the chat template is the sharp edge. gufo has no Jinja engine at all — src/models/qwen/chat_template.cpp is 587 lines of hand-written C++ that appends strings, with no --chat-template-file, no env override, and no template body to point at. scripts/qwen-sharp-v22.4.0.jinja therefore cannot be supplied to frank-large: that is a port, not a config line. Given the choice on 2026-09-28 the decision was to ship on gufo's stock renderer and accept the loss — the terse system-prompt injection, the in-band <|think_*|> control, the json|xml tool_call_format switch, the tool-argument truncation and add_vision_id prefixes are all gone. gufo's own renderer does keep the effort instruction strings and preserve_thinking, which are what the harness depends on. This is the thing to watch: the endpoint now runs an abliterated model without the prompt that tells it to stop waffling. The cheap fix, if verbosity shows up, is to send the terse text as an ordinary system message from models.yml — not a template change. reasoning_effort accepts the full OpenAI set (all six levels verified 200), so models.yml's reasoningEffortMap fold is no longer required — it is kept deliberately, so that removing it does not silently move every high role from xhigh to high effort inside a swap. And there is no KV-quantization or prompt-cache-RAM flag; prompt reuse is on by default and --cache-disk is opt-in and off.

Read the perf trade before assuming a win. gufo's own Q8_K_XL tables (vs llama.cpp b11069, DFlash2) show prefill +37 % to +80 %, cold load 4.5× faster, decode +7 % at zero depth and +40 % at 4 users — but −14 % at 65,536 tokens of depth and −24 % at 131,072. On the 8-bit target the crossover is ~32k of cached context, and frank-large runs one 262,144-token conversation, so long agent sessions sit on the losing side of that line. The Q4_K_XL target shows no such regression; revisiting the quantization is a separate decision from the runtime. Peak memory is 50.58 GiB at the full 262,144 context against the iGPU's 114 GiB GTT aperture.

The serve script (scripts/serve-frank-large.sh) validates runtime → target GGUF → drafter and exits with a pointer message if any is missing, then execs gufo serve --host 0.0.0.0 --port 7770 --sessions 1 llm --model …--served-model-name frank-large --context 262144 --speculative dflash2 …. The deployer's deploy_llm_large gates on the runtime binary as well as both payload paths, and returns before it stops anything.

Fetch: scripts/download-frank-large-swift.sh (the target; per-file byte count + sha256 against the Hub LFS oid, no auth header — the repo is public and a token on the curl command line is world-readable) and scripts/download-frank-large.sh (the DFlash2 drafter, which did not move). Both take --verify-only.

Runbook: serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md. Rollback, two ways: payload only — re-fetch with scripts/download-frank-large.sh and repoint MODEL_FILE at /home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf; that 29.3 GiB target was reclaimed on 2026-09-28 (staged to ~/.AGENTTRASH/20260928-retired-models/, restorable from there), so it is no longer a one-minute flip. Engine — frank-large-unsloth-qwen38-27b-q8xl, the b11030 prebuilt at /home/dank/GitPackages/unsloth-llama-b11030/; the only way to get Qwen-Sharp back. The DFlash2 drafter was never reclaimed and still lives in Qwen3.8-27B-UD-Q8_K_XL/.

frank-small — SGLang on the RTX 3090 (Dankclaw, ACTIVE)

SGLang (latest) in a self-contained python3.12 venv (/home/dank/GitPackages/sglang-cu128/, torch 2.13.0+cu130) as a root systemd unit (repo scripts/frank-small.service, relink_united into /etc/systemd/system/), serving letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (9B dense qwen3_5 hybrid — 8 full-attention + 24 Gated DeltaNet layers — INT8 W8A16 AutoRound, 12.0 GiB, vision tower baked into the checkpoint). Port 7771.

The shape is memory-pinned, and the sharp edge is the mamba pool. The DeltaNet ssm state pool autofills to a fixed ~5.1 GB budget regardless of dtype (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are pinned: --mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32 (795 MiB). The KV pool is then pinned at 606,208 tokens (--max-total-tokens) — the engine's autofit picks 687,388 and leaves 0.53 GB, which Triton's deferred kernel loads eat and the first long request OOMs on. Four 151,552-token slots; a single request may grow to the whole pool (170k verified). Prefill runs eager (--disable-prefill-cuda-graph) — there is no headroom for capture; decode graphs are on.

No speculative decode — a checkpoint fact. text_config declares mtp_num_hidden_layers: 1, but the shipped safetensors (base BF16 and this INT8 conversion alike) carry zero mtp.* tensors; NEXTN would draft with an uninitialised 4.24 GiB ghost model at 0% acceptance. The seam gates that no --speculative-* flag reaches the engine.

The venv carries a CUDA 13.0 nvcc extracted from debs (system nvcc is 12.4) so the sgl-kernel/flashinfer JIT compiles for sm86; three local patches ride in the tree (GCC-16 shim, CCCL symlink, rsqrt noexcept). Details in the runbook.

Verified at deploy (2026-10-01): /v1/models reports frank-small; chat returns content plus reasoning_content (SGLang's qwen3 parser uses llama.cpp's field name); a red-square image answers "Red" (image_tokens: 64); 4 concurrent 54k-token requests all answer; a 170,019-token prompt prefills in 50.5 s (~3,370 t/s eager). Decode 83.9 tok/s. Runbook: serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md. Rollback: frank-small-gemma4-12b (payload and binary intact on disk).

frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE)

The tcclaviger RDNA4 vLLM fork (docker.io/tcclaviger/vllm@sha256:da4029666…, digest-pinned) serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards) on both Radeon AI PRO R9700s (gfx1201), TP=2, MTP4 speculative decoding, expert prefetch OFF, 200,000 ctx x 3 seqs, fp8 KV. Deployed 2026-10-01 by rollback from commit 2339fae after the GLM/DFlash2 experiment was declared failed (same day).

Measured 2026-09-29 tuning campaign: decode 37.4-40.6 tok/s, acceptance ~54 % (MTP4 + prefetch off — the fleet's only endpoint that meets the 35 tok/s target). 256k ctx arm measured 34.1-36.5; 2-way concurrency sums BELOW parity (25.49 vs 26.8 single) — slots are a capacity feature, not a throughput one. V-only q8 KV is NOT possible via config: stock --kv-cache-dtype is K+V together; a V-only quant needs a fork patch.

The engine auto-sets the attention block size (mamba-page floor); the expert plan's registry prices KV per token; the expert arena (6,574 slots x 2.344 MiB = 15.05 GiB/rank) is the decode-critical region — cutting it for KV headroom collapses the offload hit rate.

Unit and self-healing: root systemd unit (scripts/frank-primary.service, User=dank, LimitMEMLOCK=infinity), ExecStop=podman stop -t 120 frank-primary-glm. The serve script drives podman run directly, gates the digest-pinned image, the payload's config.json and its exact 131-shard count and the GPU device nodes, clears stale container identities, then blocks on podman wait. deploy_llm_primary gates the same, waits out :7770, relinks/enables/restarts the unit and health-gates with the MainPID check. Boot cost ~7 min warm; health returns 503 while loading.

Runbook: serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md. Retired GLM experiment: serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md.

Key Lessons Learned

Durable operational lessons from the fleet's history. Full detail in the linked runbooks.

PCIe / multi-GPU

• RCCL collectives need PCIe atomics on every participating GPU. RCCL's kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch them when the device has no atomics, and the failure surfaces as hipErrorIllegalState ("the operation cannot be performed in the present state") from hipLaunchKernel, not as anything that names atomics. The value is consumed at driver probe time, so writing the AtomicOp Requester bit with setpci at runtime changes nothing. Check the chain: lspci -vv -s <gpu> and every upstream bridge must show AtomicOpsCap: Routing+ (a Routing- bridge is a hardware wall), and the endpoint's Device Control 2 (setpci -s <gpu> CAP_EXP+28.w) must have bit 6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the chipset (usually not) — that asymmetry is what blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL short-circuits at world_size=1.
• That wall was physical, and it was fixed on 2026-09-20 by a board swap. Both paths that put two cards on CPU root ports were: an M.2→PCIe x4 riser into a CPU-attached M.2 slot, or a board whose CPU lanes bifurcate to x8/x8. dankserver took the second: ASUS ProArt B850-Creator WiFi NEO, PCIEX16_1 + PCIEX16_2 at x8/x8 Gen5 on root ports 00:01.1 / 00:01.3, every bridge now Routing+ and both endpoints ReqEn+ — the exact precondition the old test plan named. Bandwidth was never the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4 and under 0.4 ms at Gen5 x8). Proven on the same day. vLLM's R4D one-shot allreduce runs TP=2 here and frank-medium serves on it; llama.cpp's RCCL path was built too (GGML_HIP_RCCL=ON) and its tensor mode measured — real collective, slower shape. Note the collective choice matters: TheRock's RCCL builds lack gfx1201 device code (invalid device function), which is why R4D is the path vLLM uses at all. Evidence matrix and the measured three-arm verdict: Dual-R9700-Cooperation.
• When a GPU collective dies, read the runtime, not the framework. The message that identified the cause came from AMD_LOG_LEVEL=3 (Pcie atomics not enabled, hostcall not supported → AQL dispatch failed!), not from vLLM or RCCL. AMD_LOG_LEVEL costs one env var and prints the failing HIP call with its return code.
• The software path that puts both cards on ONE model without a collective: llama.cpp's layer split. On the old board vLLM could not do it in any of its parallelisms — TP needs an allreduce, PP needs the same p2p primitive, and both died on the card that could not dispatch an RCCL kernel. --split-mode layer in llama.cpp needs no collective: layers are distributed in order and only the boundary activation crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill / 18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly fallback). This was what frank-medium ran until 2026-09-20, when the board swap made collectives legal and radiance TP=2 beat it on every axis (95.8 vs 45.7 t/s decode, 0.21 vs 0.72 s TTFT, 259 vs 72 t/s burst-4) — see frank-medium-radiance-mxfp4; the llama shape is the rollback target frank-medium-llamacpp-dflash2.
• Layer split is also the cross-vendor answer — and -ot is what cannot be combined with it. On Dankclaw (2026-09-16) a CUDA 3090 + Vulkan iGPU layer split ran a real two-device graph: -dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85 held 14,994 MiB on the 3090 and gave 27.2 tok/s with 268.5 t/s prefill at 64k depth (iGPU alone: 25.8 / 231.8). What does NOT work is per-tensor placement across devices: --override-tensor <group>=<device> sets has_tensor_overrides(), a REQUIRED term of the scheduler's pipeline_parallel flag (llama-context.cpp:433 — the only place in the tree that reads it), and that flag is what sets n_copies in ggml_backend_sched_new; with n_copies == 1 a pre-allocated weight on a non-primary device has no backend and the graph aborts (ggml-backend.cpp:941, reported with the leaf's own op name (NONE)). =CPU is the only override target that works, because the CPU backend is last in the list. Pure Vulkan changes nothing here — the gate is the override, not the driver mix. See x4-pair-experiment.
• A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail. pcie-link-test.cu on Dankclaw (2026-09-16, services stopped): pinned H2D 6.72 GB/s, D2H 6.60 GB/s — 85 % of the Gen4 x4 ceiling — with async no better than sync and pageable within 1 % of pinned, i.e. the copies are link-limited rather than staging-limited (the usual "pageable costs 2x" rule does not apply here). The card is x16-capable and the Strix Halo port beneath it is x8-capable (LnkCap: 16GT/s, Width x8 on 00:03.1), so the x4 lives in the adapter or cable; the link idles at 2.5 GT/s x4 and trains up under load. Consequence for any engine that streams weights across it: a 3 MiB slab costs 474 us, and a 111 GiB A3B model needs ~1.70 GiB of active experts per token ⇒ 3.7 tok/s (4.6 with a 20 % resident cache), against 20.46 tok/s for the same model with the experts computed in host RAM by llama.cpp (-ot exps=CPU, whence the link carries activations only) and 25–26.5 on the iGPU. Streaming over PCIe loses; computing in RAM does not care. FreeToken — expert LRU in host RAM, exactly this streaming shape — also loads only modelopt block-FP8 safetensors for qwen3_5_moe, so a Q4_K_M GGUF is unloadable there before bandwidth is even the question. See x4-pair-experiment.

ROCm / GPU

• Layer split was the deployed shape on both multi-device llama.cpp services, for two different reasons. On Dankserver it was the only way both R9700s served one model (no collective could be dispatched on card #2) until the 2026-09-20 board swap; frank-medium now runs radiance TP=2 instead. On a RCCL-less fork build it is also the fast one: tensor parallel falls back to a butterfly allreduce and collapses prefill ~7x (measured 271.8 vs 1991 t/s). Pin the spec-decode drafter to one device — an unpinned draft under tensor mode aborts the KV build, and on a layer split the natural home is the card that ends up holding the target's last layers.

• **Bundling a ROCm runtime next to a llama.cpp binary: cp -r SRC DST/ is a

  trap.** With a missing DST, cp creates DST as a copy of SRC and drops

  one directory level. rocBLAS and hipBLASLt look for their device code at a

  fixed depth (../lib/rocblas/library, ../lib/hipblaslt/library/<arch>),

  and the flattened copy fails at request time, not load time: the engine

  reads the weights, answers /health and dies on the first completion with

  `Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch

  gfx1201`. Declare the destination, then copy the contents.

• mmap must stay on for the big-PLE models: --no-mmap forces the ~50 GiB PLE table into anonymous memory → silent cgroup kill, dmesg is the only evidence. mmap pages ARE page cache. Never pin the PLE tensor with -ot (23 → 13.4 t/s).
• HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE. Serve scripts export env files with set -a, so a tunable named TEMP leaks into HIP (TEMP=1.0 sent the JIT cache to path 1.0 and segfaulted every load, fixed 2026-08-13 by renaming to TEMPERATURE). Never name a tunable these; check with strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE".
• -DGGML_CUDA_FA_ALL_QUANTS=ON is required for q8_0 KV + flash attention on CUDA builds; without it prefill collapses ~12x (95 vs 1300+ t/s, measured 2026-08-17).
• Flash-Next MTP heads do not load on any current loader — they need full per-layer blk.<il>.hc_attn_norm.weight and ship only blk.48 (verified 3x). Don't re-attempt without new loader support.
• QSA flash-attn assert: GGML_ASSERT(a-type == GGML_TYPE_F32) on qwen4exp needs scripts/patches/qwen4exp-qsa-f32-mask.patch — but only on the agentionai FAST layer set; the kingjones STRIX set works unpatched.
• GPU budget on Dankclaw is the GTT aperture (~114 GiB), not the rocm-smi "dedicated" figure. The Radeon 8060S (gfx1151) reports 116,732 MiB (llama-server --list-devices; gtt_total=122402963456 in /sys/class/drm/card1/device/). The older "64 GiB, SVM-backed" lesson was already stale — the CIRU Orca predecessor held 92.6 GiB of weights on the iGPU. free/dmidecode show the CPU-side carve-out — intentional, do not change.

Load / crash diagnosis

• Model load is slow (minutes cold); health returns 503 while loading; do not kill the process. deploy.sh waits up to 20 minutes (LLM_WAIT). On halogen, the FIRST request after a (re)start can go silent ~45–60 s (health engine_unresponsive, 503s) and then self-recovers — do not restart over it.
• Crash-loop procedure: (1) journalctl -u <svc> -n 100 + kernel line via journalctl -k (segfault at address 8 in libamdhip64 = NULL unref in HIP); (2) run serve script, then raw binary — different outcome isolates the env file; (3) gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-<svc>.sh — llama-server buffers stdout and loses the log tail on SIGSEGV; trust gdb, not the log; (4) amdgpu: SVM mapping failed spam is a symptom, not a root cause.
• Journal signatures: client abort logs W srv stop: cancel task, id_task = N; natural stop logs only slot release with truncated = 0. The server never preempts a running generation; excess tasks queue when slots are busy.

Host memory / swap

• Swap resize ordering: create and swapon the replacement BEFORE swapoff of the old one, or live pages must land in RAM that MemAvailable overstates. Dankclaw swap (64 GiB at /swap.img, fstab-driven) protects the co-tenants, not frank-large — pinned weights don't swap, and paging the KV pool is worse than the OOM it prevents.
• Host swap state is invisible to deploy.sh (no repo file) — check swapon --show before reasoning about headroom.

Runtime-specific guards

• Kairic Edge (if ever re-deployed): the pinned CK commit needs third_party/composable_kernel/include/ck/config.h generated manually (untracked; git clean removes it), and its greedy-argmax fastpath rejects non-greedy requests and reasoning budgets. Re-test chat before production.
• FastMTP sidecar: build must be commit 4df29be4 + the HauhauCS patch; stock binaries fail the draft load with a tensor-shape mismatch.

llama.cpp serving

• MTP is a property of the FILE, not of the flags. The -mtp GGUF build carries the head as blk.<n>.nextn.* (qwen35.nextn_predict_layers = 1); --spec-type draft-mtp then sets load_mtp, and the draft context runs on the MAIN model's weights (shares_model), so there is no second copy of a 12 GiB model in VRAM and no -md argument. The plain build has no MTP tensors at all, and pointing --model-draft at the file would load it twice.
• What can be chained, and what cannot. --spec-type a,b builds every enabled implementation and chains drafting through them, in a fixed internal order (ngram types first, then the drafters — CLI order is irrelevant). The gate is ctx_dft: draftless types (the ngram family) are added unconditionally, while draft-mtp, draft-dflash, draft-dspark and draft-eagle3 all require a draft CONTEXT. So a chain may hold one context-backed drafter plus any number of ngram types — draft-dflash,ngram-mod is legal and measured (2.6x the drafter alone on prompt-echo traffic, byte- identical to it on prose at the fleet's 48/64 window) — but draft-mtp,draft-dflash dies at load: "context type MTP requested but model doesn't contain MTP layers", because MTP builds its context against whatever model the draft context holds. Two context-backed types cannot share one process, and MTP's head only loads when MTP is the draft model.
• A drafter's KV scales with the TARGET's context, and its window policy decides how much that costs. The draft context is built with cparams.n_ctx = llama_n_ctx(ctx_tgt) (common/speculative.cpp) — there is no flag to cap it — so a drafter that declares NO sliding window caches the whole window on whatever device --spec-draft-device names. Swapping DFlash2 (dflash.attention.sliding_window = 2048, ~40 MiB of draft KV) for DSpark (no window; 5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per position, so 5.0 GiB at 262144) put ROCm1 at 33.9 GB of 34.2 GB and killed the FIRST draft graph capture with ROCm error: out of memory — after the model had loaded and /health had answered. Read sliding_window out of the drafter's GGUF header before sizing any spec-decode deployment, and prefer a skewed -ts over a context cut when the full window matters. The pair was swapped DFlash2 -> DSpark -> DFlash2 in one day (2026-09-16) and the round trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 -> 36:29), which is the practical size of the effect. See frank-medium-llamacpp-dflash2.
• --ubatch-size has an OPTIMUM on this stack, not a plateau — and its compute buffers land about TWICE as heavy on the card that holds the output layers and the drafter. Measured on frank-medium 2026-09-16 at the deployed shape (262144 over two slots, a 64k prefill measured at 64k depth, scripts/bench-batch-ubatch.sh): prefill runs 1046 tok/s at ubatch 256, 1068 at 384, 1193 at 512 (the peak), 1135 at 768, 1152 at 1024, 1151 at 1280, 1165 at 1536, 1148 at 1792 and 1078 at 2048. Both ends lose — below 512 the SMs are not saturated, and 2048 is 10 % off the peak — and 512 -> 1536 costs ROCm0 1.30 GiB and ROCm1 2.41 GiB to go 2.4 % SLOWER. The logical --batch-size does not matter: 2048 through 8192 land within 2 % of each other at a fixed ubatch and use identical VRAM. Decode never moves at all (15.88 tok/s at depth 65536 on every cell). Because the buffers are asymmetric, a split balanced at one batch is unbalanced at another: 512 -> 1536 shifted frank-medium's balance by a layer, so 36:29 became 35:30. Re-measure the balance after changing the batch. (8192/2048 failed to allocate its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had 0.08 GiB free. That was a CONTIGUITY failure, not a total-bytes one, and no totals model — the sweep's or the seam's — can see it.)
• One -ts step is one LAYER, and a hybrid stack's layers are not the same size. qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one (0.45 GiB of weights plus ~1.0 GiB of its own f16 KV at 262144) — both measured on frank-medium. Balance by measurement, never by interpolating between two splits; an affine fit reproduces the deployed point and overstates the gaps in between. tests/frank_medium_unit.bats encodes that fit as a gate. See frank-medium-llamacpp-dflash2.
• ngram-mod prints no startup banner. The load log shows the drafter's adding speculative implementation 'draft-dflash' and nothing at all for the matcher, so a chain can look unengaged while it is working. Identify it behaviourally (draft_n > 0 only where text repeats) or by the per-task draft acceptance = ... mean len = ... line. It costs no VRAM (a 64 MiB host-RAM hash table) and its window decides which traffic it can see: too short and it proposes on ordinary prose fragments, which costs verification without acceptance.
• --jinja is what tools needs. On 2026-09-16 a subagent died with 500 tools param requires --jinja flag, from the ik_llama.cpp runtime that then served frank-large: every tool-using agent on that endpoint was broken and the failure read like a model fault. Mainline llama.cpp 0.4.1 defaults --jinja on and the serve script pins it; halogen also accepts tools and returns tool_calls. Check tool support before routing a role at an endpoint.
• Quantized KV on CUDA needs GGML_CUDA_FA_ALL_QUANTS=ON — the same option and the same ~12x prefill collapse described in the ROCm note above. f16 KV needs nothing; frank-small ran f16 for that reason until 2026-09-17, when build-cuda was rebuilt with the option ON so the endpoint could take q8_0 KV.
• llama-bench separates a LIST with / and SWEEP values with ,. Both -dev and -ts follow this: -dev CUDA0,Vulkan1 is two single-device configurations, never one two-device run, and -ts 0.85,0.15 is two sweep values each becoming [0.85, 0, …] / [0.15, 0, …], i.e. device 0 takes everything. Write -dev A/B -ts x/y. This artifact produced a wrong fleet conclusion on 2026-09-16 — "CUDA + Vulkan cannot serve one model", with two bogus OOMs that were really "all 111 GiB on the 24 GiB card" — and the bench table's own dev column was printing the truth the whole time. Any placement-dependent number needs an independent check: sample the other device's memory (nvidia-smi) while the run is loaded.
• A model's own template can REJECT the harness's thinking level. Qwen3.8's template accepts only xhigh, medium and low for reasoning_effort and raises on anything else — HTTP 500, not a clamp — so the harness's levels must be folded onto that set in models.yml (compat.reasoningEffortMap). A role whose level is unmapped fails on its FIRST request, and the failure names Jinja, not the model: every subagent on frank-small died with Unexpected reasoning effort high until high: xhigh was added on 2026-09-16. Halogen accepts the wider set, so the two entries legitimately differ.

Ops

• Never pkill -f a pattern that appears in your own command line — it kills your ssh session. Use pkill -x <exact-name>.
• Both hosts allow passwordless sudo for the deploy user. Use plain sudo in committed scripts (no echo pw | sudo -S); FRANK_SUDO_PW covers hosts without it.
• ComfyUI on Dankclaw runs with GPU access (/dev/kfd + /dev/dri via AddDevice, GroupAdd=keep-groups, plus the udev rule that makes those nodes world-writable — user-unit spawns carry no supplementary groups). It must NOT pass --cpu: the gfx1151 ROCm wheels exist and the unit's --cpu era ended 2026-09-16, when a stale device-less container was found restarted 9,146 times (RuntimeError: No HIP GPUs are available).
• SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its ports off the gluetun service.
• amd_iommu=off on Dankclaw (bought 13–16 % prefill for frank-large, 2026-09-11) permanently retired the XDNA2 NPU: the driver's SVA/PASID bind needs an enabled IOMMU (/dev/accel/accel0 fails err=-19). Revert path in serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md.
• Rootless Quadlet containers run with --replace --rm: podman logs covers only the CURRENT instance; history across recreations lives in journalctl --user -u <name>.service. Read rootless storage WITHOUT sudo (per-user storage — sudo podman logs finds nothing).
• A dead rootless podman pause process breaks ALL container management on the host, and podman system migrate panics in 5.7.0. Symptoms: every podman command fails with invalid internal status … could not find any running process, every container start fails binding podman's bridge addresses (Error starting server failed to bind udp listener on 10.89.x.1:53: Cannot assign requested address), and every Restart=always quadlet loops. Container storage is intact; what is gone is /run/user/<uid>/libpod/tmp/alive (a UUID token, not a pid) and the rootless netns holding the bridges — deleting the stale token is not enough, and podman system migrate segfaults in teardownNetworkBackend. Repair: reboot (/run/user clears and all 28 quadlets return on their own — check Linger=yes and WantedBy=default.target first). Seen 2026-09-16, hours into heavy GPU work on the same host — but the trigger is NOT established (journalctl -k recorded no OOM kill; the kernel had no other word on it). Treat the pause process as host state that can die for its own reasons, and reach for the reboot when it does.
• A container cannot reach a Network=host service at the host's LAN IP. netavark DNATs published ports only: from a bridge container 192.168.0.143:<published> answers while 192.168.0.143:<host bind> and the gateway 10.89.1.1 are both refused. Use host.containers.internal (169.254.1.2). This is what 502'd hermes.franksdank.cloud until the Caddyfile upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP was open from that bridge — it never was).
• A *.gguf filter misses mradermacher's payloads: large quants ship as .partNofM fragments. mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF carries Q4_K_M (37.00 + 37.00 + 36.97 = 110.97 GiB) and Q4_K_S (35.00 + 35.00
  • 34.09 = 104.09 GiB) as parts, while the smaller rungs (Q2_K, Q3_K_, IQ3_, IQ4_XS) are single files. Fetch the parts, verify each against the Hub's LFS sha256, concatenate in numeric order — that is how the deployed 111 GiB file was assembled. Listing the Hub tree and filtering endswith(".gguf") makes Q4_K_S look like it does not exist.
• A fetch script must -f and verify, or it reports success on an access error. curl -o FILE URL exits 0 on HTTP 401/403 and writes the error body as the "artifact": a gated repo yielded three 153-byte shards with identical sha256 values and a cheerful DOWNLOAD_DONE in 20 seconds for 104 GiB. Use -f/--fail-with-body, compare each file's byte count to the Hub's declared size, and check the LFS sha256 — scripts/download-frank-medium-gguf.sh already does all three; ad-hoc fetches tend not to. Gated repos (e.g. orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF, same 104.09 GiB Q4_K_S) return 403 even with the host's HF token until that account accepts the gate.

Runbook Index

Active:

Service	Runbook
frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw)	serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md
frank-small (SGLang MiMo-V2.6 INT8, Dankclaw)	serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md
frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver)	serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
Dankclaw kernel posture (amd_iommu=off)	serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md
Deployment flow + incidents	serverAppConfigs-docs/LLM/deployment.md

Retired (rollback targets and history):

Runtime	Host	Window	Runbook
tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (frank-primary)	Dankserver	2026-09-30 → 10-01	frank-primary-glm53-flash-w4a16.md
llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (frank-large)	Dankclaw	2026-09-21 → 09-28	frank-large-unsloth-qwen38-27b-q8xl.md
R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (frank-primary)	Dankserver	2026-09-27 (same-day swap to Q8_0)	frank-primary-r9v-radiance-qwen38-uncensored.md
llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (frank-primary)	Dankserver	2026-09-26 → 09-27	frank-primary-llamacpp-qwen38-flash-next-q8.md
ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (frank-large)	Dankclaw	2026-09-15 → 09-16	frank-large-ik-uncensored.md
vllm-radiance TP=2 MXFP4 + DFlash2 (frank-medium)	Dankserver	2026-09-20 → 09-26	frank-medium-radiance-mxfp4.md
Lucebox DS4.1 DeepSeek-V4.1-Flash ROCmFP23 (frank-primary)	Dankserver	2026-09-26 (same-day swap)	frank-primary-lucebox-ds4.md
kingjones STRIX Qwen3.8-Flash-Next ROCmFP4	Dankclaw	2026-08-30 → 09-11	kingjones-strix-rocmfp4.md
halogen-flash-server Qwen3.8-Flash-Next w4b (frank-large)	Dankclaw	2026-09-16 → 09-19	halogen-flash-server.md
CIRU v4.4.1 Orca Qwen3.8-Flash (frank-large)	Dankclaw	2026-09-19 → 09-21	frank-large-ciru-orca.md
unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (frank-large)	Dankclaw	2026-09-21 (same-day swap)	frank-large-unsloth-qwen38-flash-next.md
Kairic Edge Qwen3.8-27B IU4 W4A4 (frank-large)	Dankclaw	2026-09-21 (same-day swap)	frank-large-kairic-edge-qwen38-27b.md
ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (frank-small)	Dankclaw	2026-09-16 → 09-17	frank-small-qwen38-gsq-rco.md
vLLM Gemma4-12B w4a16 (frank-small)	Dankclaw	2026-09-19 (same-day revert)	frank-small-vllm-gemma4.md
llama.cpp Gemma4-12B QAT Q4_K_M (frank-small)	Dankclaw	2026-09-17 → 10-01	frank-small-gemma4-12b.md
agentionai ROCmFP4-FAST imatrix	Dankclaw	2026-08-29 → 08-30	agentionai-fast-imatrix.md
eemin Qwen3.8-27B ROCmI4 W4A4 MTP	Dankclaw	2026-08-18 → 08-25	eemin-rocmi4-mtp.md
Kairic Edge IU4	Dankclaw	2026-08-22 (same-day revert)	kairic-edge.md
CIRU ActiveFPX v3 / PromptForge	Dankclaw	through 2026-08-22	ciru-activefpx-promptforge.md
DFlash2 Vulkan	Dankclaw	2026-08-21 (same-day revert)	dflash2-large-vulkan.md
FastFlowLM NPU (frank-npu)	Dankclaw	twice, retired 2026-09-11	frank-npu.md, FastFlowLM NPU Inference.md
FreeToken (official + Huihui FP8, frank-small)	Dankserver	2026-09-02 → 09-15	freetoken-frank-small.md
FastMTP patched llama.cpp	Dankserver	retired 2026-08-18	qwen3.8-spec-matrix.md
ik_llama.cpp Ornith CRACK	Dankserver	retired 2026-09-02	ikllama-ornith-dankserver.md
llama.cpp layer split + DFlash2 (frank-medium)	Dankserver	2026-09-16 → 09-20	frank-medium-llamacpp-dflash2.md

Retired runbooks are under serverAppConfigs-docs/LLM/. Retired scripts live in old/ and .AGENTTRASH/.

Why

Direct edits on the remote drift from version control, and future agents won't know what changed. Git is the single source of truth. The two workloads share two hosts, one Caddy, and one Gitea; one repo keeps them in one history.

Summary

Context	Action
Editing a compose/Caddyfile/serve script	Through git — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote.
Config exists only on remote	Pull it into the repo first — commit the fetched version (redacted) before editing.
Unstaged local changes	Commit or discard — check git status before starting work; ask if unsure.
Deploying a committed change	scripts/deploy.sh <target> — pushes, pulls ff-only, plans, dispatches, health-checks.
Container without a Quadlet unit	Give it one — boot-persistent, self-healing .container via git.
Benchmarking	Stop and mask the service first, bench, unmask, restart, verify health.
Deleting a file	Move it to trash — .AGENTTRASH/ (in-repo) or ~/.AGENTTRASH/YYYYMMDD-topic. Never rm.
Tests exist	Run them — verify tests pass after every change. Do not commit failing tests.
Real secrets	Never commit — host .env files; .env.example holds placeholders only.