serverAppConfigs/AGENTS.md

AGENTS.md — serverAppConfigs Agent Rules

Overview

This repository is the single source of truth for the two homelab servers and the FrankBook-Pro Mac mini. It tracks the Podman container stacks and the LLM inference fleet. One repo, one checkout per host, one deploy script.

• Dankserver (192.168.0.235) — media/library stack (RomM, Copyparty, Calibre-Web, Jellyfin, Plex) as bare containers, plus the frank-small LLM service.
• Dankclaw (192.168.0.143) — AI/search/development/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, Alloy, Pi-hole) plus the frank-large LLM service. Caddy on Dankclaw reverse-proxies both hosts.
• FrankBook-Pro (192.168.0.112) — macOS mini running the *arr stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside its podman machine; every container rides the Gluetun WireGuard netns, so all *arr and qBittorrent egress is VPN-tunneled. Deployed via the frankbook-pro handler.

2026-08-22 merge: the former vllmScripts repository (LLM fleet scripts, units, env tunables, and runbook) is merged into this repo. Its full git history is preserved (two-parent import commit under the retired llm/ prefix, reorganized in follow-up commits). The retired target checkouts (/home/dank/vLLM on Dankserver, /home/dank/vllmScripts on Dankclaw) are archived. All services now run from /home/dank/serverAppConfigs on each host.

LLM fleet topology (2026-09-11)

Target	Runtime	Service	Model	Port
Dankserver	FreeToken v0.1.2	frank-small	batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on	7770
Dankclaw	halogen-flash-server 0.5.8 (rootless Quadlet container)	frank-large	peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 262144, vision tower on, greedy for silent clients)	7770
Dankclaw	(retired 2026-09-11) FastFlowLM v1.0.1 on the XDNA2 NPU	frank-npu	Gemma4-E4B-IT-NPU2 pack (vision + audio). Retired with the amd_iommu=off decision: the driver's SVA/PASID bind needs an enabled IOMMU	7771

History: the 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw. On 2026-08-21 a DFlash2 Vulkan deployment briefly replaced CIRU, then was reverted the same day for speed. On 2026-08-22 the service briefly swapped to the Kairic Edge IU4 runtime, then was reverted to v3 the same day (operator decision). On 2026-08-25 the service switched to the kingjones Qwen3.8-27B-Uncensored-Q6_0-ROCmFPX_AGENT model (standalone MTP draft, 262144 context, reasoning on). On 2026-09-01 both services were swapped: Dankserver switched from ik_llama.cpp Ornith CRACK Q8_0 to stock llama.cpp Tiel-Coder-35B (qwen35moe hybrid MoE), and Dankclaw switched from the Q6_0 AGENT model to the Flash-Next 56B MoE (qwen4exp, PR #27742 merged into charlie's ROCmFPX fork, --fit off, 128k context). On 2026-09-02 Dankserver moved to the FreeToken runtime serving the official Qwen3.6-35B-A3B-FP8. On 2026-08-30 both services swapped again: Dankclaw to the kingjones STRIX model on the card's pinned kingjones30/ROCmFPX runtime, Dankserver to the Huihui abliterated FP8 checkpoint on FreeToken. On 2026-08-30 the NPU endpoint returned as frank-npu — FLM v1.0.1 on XDNA2 serving sight+sound. On 2026-09-11 Dankclaw's large service swapped runtime and model: the kingjones STRIX llama.cpp deployment gave way to halogen-flash-server 0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next, now a rootless Quadlet container, and the kernel cmdline moved from iommu=pt to amd_iommu=off for 13–16 % of prefill — which retired frank-npu for good (see the Runtime & Troubleshooting section and its runbook). Runbooks: serverAppConfigs-docs/LLM/index.md. Retired scripts live in old/ (reference only) and .AGENTTRASH/.

Repository Layout

• dankserver/ — compose.yaml + Caddyfile + .env.example for the media stack
• dankclaw/ — compose.yaml + Caddyfile + .env.example for the AI stack
• shared/ — homepage, Prometheus, Loki, Alloy, SearXNG configs
• scripts/ — deploy.sh, deploy-plan.py, and deploy-remote.sh (service-scoped deployer), LLM serve scripts, systemd units, env tunables, and bench harnesses
• old/ — retired PP/NPU-era LLM scripts (reference only)
• serverAppConfigs-docs/ — knowledge base (Obsidian vault). serverAppConfigs-docs/LLM/ is the LLM fleet
• tests/ — bats test suite

Documentation Authoring

The serverAppConfigs-docs/ vault is the project knowledge base (Obsidian, wikilinks) and is an OKF (Open Knowledge Format) knowledge bundle. Follow the OKF spec for every doc you create or edit.

• Canonical spec: ~/.pi/agent/docs/references/okf-spec.md — the OKF v0.1 spec in the system-config knowledge base at ~/.pi/agent/docs. The repo's own serverAppConfigs-docs/references/okf-spec.md is deprecated/removed; use the system one. The spec index and maintenance guide are ~/.pi/agent/docs/references/index.md and ~/.pi/agent/docs/references/okf-maintenance.md.
• Read before you touch: open the relevant index.md first (serverAppConfigs-docs/index.md → serverAppConfigs-docs/LLM/index.md). Read the whole target file before editing it.
• One concept per file, with YAML frontmatter that has a non-empty type: field (reference / guide / configuration / research) plus description, tags, and timestamp. Split broad topics; keep pages ~1–500 lines.
• Progressive disclosure: an index.md at each level links its children. Every serverAppConfigs-docs/**/*.md (except index.md) must be linked from serverAppConfigs-docs/index.md with [[Title]] — the docs-completeness test checks this.
• Cross-link siblings as [[Title]] wikilinks; cite external claims under a # Citations section.
• Always update docs when a task changes behavior, then run ./tests/run.sh (the docs-completeness seam is in there) before commit.

Git Workflow

All edits flow through git. Do not edit files on the remote machine directly.

Protocol

1. Source the config from the target machine — fetch the live compose/Caddyfile/serve script from the server where it runs
2. Place it in this repo — commit the fetched version (redact secrets into .env)
3. Edit here — make all changes locally in the working tree
4. Run the tests — ./tests/run.sh must pass
5. Commit and push — push to origin (Gitea on Dankclaw)
6. Pull on the target — git pull on the remote machine
7. Deploy — scripts/deploy.sh pulls, plans, and dispatches only the changed service(s), then checks the selected service health

Deployment

Use scripts/deploy.sh to deploy committed changes to a target. The planner maps changed paths to individual services; the remote runner restarts only the selected service and records its successful revision outside the checkout.

./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankserver
./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-small
./scripts/deploy.sh dankclaw --dry-run

A documentation-only change is a no-op. The LLM services are selected only by their own unit/env/serve files or the shared chat template, so unrelated container changes cannot interrupt inference. frank-large is selected by its Quadlet file under dankclaw/frank-large/quadlet/.

Set FRANK_SUDO_PW for headless runs on a host without passwordless sudo (remote sudo reads it from stdin via sudo -S). The value is never stored in this repo.

Deployment completeness (MANDATORY)

A deployment is not complete when deploy.sh exits green. It is complete only when BOTH of the following pass:

1. Unit/config tests — ./tests/run.sh (all seams green), and
2. Playwright end-to-end tests — tests/e2e/run.sh drives the LIVE fleet (homepage, Anubis/Authentik guard chain, guarded vhosts) in a real browser. The E2E suite is deliberately excluded from ./tests/run.sh because it needs live hosts and credentials, so it MUST be invoked explicitly after every behavior-affecting deploy.

graph LR
    A[Edit locally] --> B[Tests pass]
    B --> C[Commit]
    C --> D[Push to origin]
    D --> E[deploy.sh computes service plan]
    E --> F[Target pulls with ff-only]
    F --> G[deploy-remote.sh dispatches selected service]
    G --> H[Per-service health check]

Boot Persistence & Self-Healing (MANDATORY)

Every Podman container governed by this project that is successfully deployed and proven working MUST have a self-healing, boot-persistent Quadlet unit. A container that runs only from compose.yaml (or a manual podman run) dies silently at reboot: rootless Podman does not auto-start bare containers, and restart: unless-stopped does not survive the explicit stop that shutdown performs. Do not report such a service as "up" without its unit.

Each container's unit must provide:

• Boot persistence — [Install] WantedBy=default.target, installed to ~/.config/containers/systemd/<name>.container (generating <name>.service) and enabled in the user manager (systemctl --user enable <name>.service). Linger must be on for the deploy user (loginctl show-user dank -p Linger → Linger=yes) so user units start at boot with no login session.
• Self-healing — [Service] Restart=always (or Restart=on-failure) with a short RestartSec, so a crash or OOM brings the container back without operator action. Add a quadlet health check where the service has one.
• Repo provenance — the .container file lives in this repo (dankclaw/<svc>/quadlet/ or dankserver/<svc>/quadlet/), is installed by the service-scoped deployer, and is covered by the quadlet test seam. The host-side unit is generated state; the repo file is the source of truth.

Verification before a service is declared deployed: systemctl --user is-enabled <name>.service → enabled, systemctl --user is-active <name>.service → active, and the container answers its health check. A running container with no unit is a gap: write the unit, test it, deploy it.

Incident that established this rule (2026-08-26, Dankclaw reboot): Caddy ran for weeks as a bare compose container and never came back after reboot; the Pi-hole unit existed but crash-looped on a sysctl conflict (a later-sorted drop-in reset net.ipv4.ip_unprivileged_port_start to 80), taking LAN DNS down with it.

Known gap (2026-09-11, found while rebooting Dankclaw for the amd_iommu=off change): five Dankclaw services still run only from dankclaw/compose.yaml with no Quadlet unit — prometheus, grafana, loki, alloy and node-exporter (all under ~/.config/containers/systemd/ absent, but present in the compose file). They did not come back after that reboot and were restored with ./scripts/deploy.sh dankclaw --service <name>; they will not survive the next one either. Giving them units is tracked work, not a solved problem — treat every reboot as expected to take the observability stack down until it is done.

Testing

Tests are mandatory before commit. bats must be installed on the development machine.

./tests/run.sh

Test seams: compose validity, env consistency (no leaked secrets), Caddyfile validity, quadlet units, docs completeness, script syntax (all of scripts/, including the LLM serve scripts), LLM unit integrity (units point at files in this repo), LLM secret hygiene, deployment flow, and service-scoped deployment planning/dispatch. See serverAppConfigs-docs/Architecture/Testing Policy.md.

Secret Rules

• Never commit real secrets. The live Gitea database password and the live sudo password are test-enforced as leak checks.
• .env files are git-ignored. Commit only .env.example with placeholders. The LLM env tunables in scripts/ are tracked by exception (!scripts/*.env). They hold paths and numbers, never credentials.
• Do not document passwords in this repo, including sudo passwords. SSH keys are the auth mechanism. Headless sudo passes the password via the FRANK_SUDO_PW environment variable. The value never enters the repo.

External Exposure Policy (MANDATORY)

Every service reverse-proxied through Caddy is either behind the auth chain (Anubis :8923 → Authentik embedded outpost) or explicitly designated internal-only — no public DNS name, no vhost. New services default to internal-only until guarded.

When guarding a new vhost, follow this order — provider first, then route:

1. Create the outpost proxy provider FIRST (external_host = the exact vhost URL including trailing slash; internal_host = http://caddy:8017). If the vhost flips before the provider exists, authenticated traffic hits the outpost error page instead of the app.
2. Flip the vhost to reverse_proxy anubis:8923 with header_up X-Real-Ip {remote_host}, and add the post-auth @<svc> host <vhost> handler on the http://:8017 listener routing to the real upstream.
3. The Anubis guard set, the Caddy TLS edge, and the outpost provider list must stay in lockstep — they change together in one deploy.
4. Service-to-service traffic (e.g., Alloy → Loki push) targets internal addresses (loki:3100), never the public hostname — non-browser clients cannot solve challenges.

Exempt-by-construction categories stay exempt with a documented reason in the vhost comment: the IdP itself (auth. — its login flow must never sit behind a challenge), git smart-HTTP endpoints (git clients cannot solve JS challenges), and services with their own complete auth.

Deletion & Trash Policy (MANDATORY)

Never run rm to delete files. The command deletes permanently. Move every item you delete into trash instead. The trash keeps the deletion recoverable and reviewable.

1. In-repo files: move the item to .AGENTTRASH/ beside its directory.
2. Home and other paths: create a dated staging directory ~/.AGENTTRASH/YYYYMMDD-topic and move each item there.
3. List the staged items and their sizes for the user.
4. Wait for explicit user permission before you empty the trash.

Rules:

• rm, rm -rf, and git clean -f are forbidden without explicit user permission.
• git * --no-verify is forbidden.
• A moved item stays in the trash until the user approves its purge.
• The trash is a temporary hold, not a long-term archive.

Benchmarking Policy (MANDATORY)

STOP the production service BEFORE running any benchmark on a target. These servers serve real users (about 3 people). Running a bench while the service is up starves the GPU and corrupts results. Restart the service when the bench finishes.

ALWAYS bench using llama-bench for llama.cpp or vllm bench for vllm runtimes. DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE OFFICIAL BENCHES. ASK THE USER BEFORE DEVIATING FROM THIS DIRECTIVE.

1. Stop the service before any benchmark: systemctl --user stop frank-large (Dankclaw — a rootless Quadlet service, so drop the sudo) or sudo systemctl stop frank-small (Dankserver — still a system unit). Then mask it to prevent the self-healing unit from restarting: systemctl --user mask frank-large (Dankclaw) or sudo systemctl mask frank-small (Dankserver). CRITICAL WARNING: Do NOT use systemctl disable on the system units. On these hosts, systemctl disable doesn't just remove symlinks — it deletes the unit file itself from /etc/systemd/system/. You'd have to manually copy the unit file back from this repo, run daemon-reload, then enable and start. Mask (systemctl mask) creates a symlink to /dev/null instead, which is safe and reversible.
2. Run the bench with the runtime's own harness: llama-bench for llama.cpp, vllm bench for vLLM, and for frank-large's halogen-flash-server the image's own bench/sweep modes (bench [drafters] [max_tokens] [effort] [reps]) — llama-bench and vllm bench do not apply to it. Scratch ports are not used for tests or benches on the LLM hosts — a test process cannot share the single GPU with the running service anyway (the model + KV fill the card), so changes are validated directly against the service itself.
3. Restart the service when the bench completes: unmask, then start: systemctl --user unmask frank-large && systemctl --user start frank-large (Dankclaw) or sudo systemctl unmask frank-small && sudo systemctl start frank-small (Dankserver).
4. Verify health before reporting: curl http://192.168.0.235:7770/health or curl http://192.168.0.143:7770/health.

Model Config (harness)

The harness reaches the fleet through /home/dank/.omp/agent/models.yml — verified live 2026-09-11; the older /home/dank/.pi/agent/models.json path this section used to name no longer exists. Both files live outside the repository. Each provider is one host endpoint and each model id is the served alias:

alias	endpoint
frank-small	Dankserver 192.168.0.235:7770
frank-large	Dankclaw 192.168.0.143:7770
gemma4-it:e4b	(removed 2026-09-11 with frank-npu)

Two resolution rules apply (verified in @earendil-works/pi-coding-agent model-resolver.js):

• The default model resolves as a defaultProvider + defaultModel pair. The defaultProvider must name the provider that serves the defaultModel.
• A subagent override uses a bare model id. A bare id resolves only when it is unique across all providers.

Per-role routing lives in /home/dank/.omp/agent/config.yml under modelRoles (smol, tiny, task, plan, designer, advisor, slow, vision, …). When you rename a service, retire an endpoint, or move a model to a new host, update the provider block, the model id, contextWindow, maxTokens and input in models.yml, then fix every modelRoles entry that named the old endpoint — a role pointing at a removed provider fails at call time. frank-large declares input: [text, image] because halogen runs its vision tower.

Runtime & Troubleshooting

frank-npu — FLM NPU endpoint (Dankclaw, RETIRED 2026-09-11)

Retired by operator decision together with the kernel cmdline change that bought 13–16 % of prefill for frank-large: with amd_iommu=off the XDNA2 driver cannot bind SVA/PASID, so every open of /dev/accel/accel0 fails with err=-19. The unit is stopped and masked, the service files are staged in .AGENTTRASH/ (*.flm-npu), and the providers that named it were removed from both the harness and Hermes. Audio input left the fleet with it and has no replacement; image input survives on frank-large through halogen's vision tower. Route map: Hermes vision → frank-large, Hermes title_generation → frank-small, harness vision → frank-large, harness tiny → frank-small, harness task → frank-large at medium effort.

The retired runbook keeps the full record, including the original IOMMU/SVA root cause: serverAppConfigs-docs/LLM/frank-npu.md. The current kernel posture and its revert path: serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md.

halogen-flash-server (Dankclaw, ACTIVE since 2026-09-11)

The large service runs ghcr.io/peonist-ai/halogen-flash-server:0.5.8 as a rootless Quadlet container (dankclaw/frank-large/quadlet/frank-large.container → generated frank-large.service in the user manager), serving peonist-ai/halogen-qwen3.8-flash-next: Qwen3.8-Flash-Next as a 115.55 GiB w4b .hgn checkpoint plus its quality overlay sidecar. One container runs both halves — the engine on container loopback :8730 (unauthenticated, never published) and the OpenAI front-end on :8731 published as host 7770 — with HALOGEN_MODEL_ID=frank-large, so no client wiring changed.

Deployed flags: HALOGEN_CTX=262144, HALOGEN_KV_POOL_POSITIONS=262144 (conservative for a shared host), HALOGEN_KV_SLOTS=4, HALOGEN_HOST_RESERVE_GIB=24, HALOGEN_VISION_TOWER=1, and HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next for the first-start fetch; everything else is upstream's default. The weights stay pinned (~68 GiB) and the kernel counts those locked pages as reclaimable page cache, so free, MemAvailable and anything built on them (Grafana) overstate free memory by the size of the model — believe the server's own startup ledger, not the dashboards.

Client-visible contract: no server-side sampler defaults (a request that sends no temperature decodes greedy); the per-request budget defaults to 8192 and includes thinking, so a budget that runs out mid-thought returns finish_reason: "length" with empty content; the cap is 65536; and a request reserves prompt + max_tokens positions against the KV pool, so a request larger than the pool is never admitted — it queues and then fails with 503 engine_busy after the 3600 s queue timeout.

Runbook: serverAppConfigs-docs/LLM/halogen-flash-server.md.

kingjones Qwen3.8-Flash-Next STRIX on kingjones30/ROCmFPX (Dankclaw, PRIOR — rollback target)

The large service served kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF (public, 113.47 GiB single file, Q4_0_ROCMFP4_STRIX = file_type 105) from /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/ with the card's pinned runtime: the kingjones30/ROCmFPX fork (charlie12345/ROCmFPX + qwen4exp-on-rocmfpx-d3ca537 patch — ROCmFP4 types and qwen4exp arch in one tree), built at /home/dank/GitPackages/kingjones-ROCmFPX/build with -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_WEBUI=OFF (WebUI provisioning needs npm/HF-bucket and aborts the link at 98% without the OFF flag). Args: our validated profile — 401408 context / 2 slots, b 4096 / ub 512 (ub 512 matches the card's re-measurement), t 16 / tb 16, f16 KV, --fit off, --flash-attn on (works UNPATCHED on this model — the QSA F16 ggml_fill assert that needed scripts/patches/qwen4exp-qsa-f32-mask.patch is specific to the agentionai FAST layer set; if a future build aborts with GGML_ASSERT(a-type == GGML_TYPE_F32), port that fix), reasoning on (Sharp template, deepseek format), no speculative decoding (card ships no spec flags; Flash-Next MTP heads do not load on this loader family). Two hard card constraints: mmap must stay on (--no-mmap forces the 50.66 GiB Q8_0 PLE table into anonymous memory → silent cgroup kill, only dmesg evidence; llama.cpp's "use_mmap=1" log line is correct here — mmap pages ARE page cache) and never pin the PLE tensor with -ot (23 → 13.4 t/s). The 51.2B-param PLE table never enters GPU memory (~20 bytes/token-file per 500 decoded). On-host file sha256 verified == HF LFS oid c6770d74...49fa. Its env and serve script (scripts/frank-large.env, scripts/serve-frank-large.sh) are staged in .AGENTTRASH/, suffixed .llama.cpp. Runbook: serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md.

Serving status (2026-08-30): swapped from the agentionai FAST deployment. Validated live before the systemd handoff: clean load, health 200, reasoning completions, and an operator session's ~111k token prefill at 213–232 t/s (card rung: 196 at 131k). Note llama.cpp clamps n_ctx_train 262144 → per-slot 131072 with parallel 2.

agentionai ROCmFP4-FAST imatrix llama.cpp (Dankclaw, prior, 2026-08-29 to 2026-08-30)

Historical — rollback target. agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB, Q4_0_ROCMFP4_FAST imatrix, v2/ joined-PLE layout only — the root -ple16 per-head split aborts this loader family) on the charlie build-qwen38 tree (/home/dank/GitPackages/ROCmFPX, PR #27742 merged) at 32768/1, ub 4096. That tree REQUIRES scripts/patches/qwen4exp-qsa-f32-mask.patch for --flash-attn on (F16-kq_mask ggml_fill abort otherwise). Flash-Next MTP heads do not load on any current loader (require full per-layer blk.<il>.hc_attn_norm.weight; heads carry only blk.48 — verified 3x). Binary and payload retained. Runbook: serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md.

Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on FreeToken (Dankserver, ACTIVE)

The small service serves batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (public, ~35 GB, 26 safetensors shards) from /home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8/ with FreeToken v0.1.2 (/home/dank/freetoken/.venv, ft serve). Native modelopt block-fp8 (e4m3, dynamic, 128×128 weight_block_size, un-fused per-expert layout) — same format and loader path as the official Qwen3.6-35B-A3B-FP8 it replaced on 2026-08-30, so the entire memory/pool plan carries: experts host-RAM with 20% LRU GPU residency, paged KV (only the 10 full-attention layers page; 30 GDN layers use constant-size mamba slots), ~3.4 GiB VRAM headroom at boot, and the validated --memory-num-gpu-blocks-experts 2048 --memory-num-gpu-tokens-kv-cache 453005 --max-num-seqs 4 pool triple (never grow the expert pool). 262144 context, reasoning on (the checkpoint ships the froggeric fixed chat template — no override). FreeToken's API REQUIRES an explicit "model" field in chat requests (422 without it). Env scripts/frank-small.env, serve scripts/serve-frank-small.sh. Runbook: serverAppConfigs-docs/LLM/freetoken-frank-small.md.

Serving status (2026-08-30): swapped the model within the FreeToken runtime (official FP8 → Huihui abliterated FP8). ~40 tok/s verified on a 282-token completion post-deploy. Prior: official Qwen FP8 (2026-09-02, runtime swap), Tiel-Coder-35B on llama.cpp (2026-09-01), ik_llama.cpp Ornith CRACK (retired 2026-09-02).

charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, prior v3, 2026-08-18 to 2026-08-25)

The v4 runtime serves cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF (eemin namespace, public) from /home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server (charlie12345/ROCmFPX commit c49ebdbd built with -DGGML_HIP_ROCMI4_W4A4=ON against system ROCm 7.2.4 / amdclang 22). The model is Q4_0_ROCMI4 with an embedded NextN/MTP head (no separate draft file); the gfx1151 native IU4 WMMA instruction batch-verifies proposals. Profile: strict Qwen MTP (--spec-mtp-strict-qwen, n_max 16, p_min 0.60, backend sampling, all layers on ROCm0), -t 16 -tb 32, batch 512 / ubatch 256 (ubatch swappable via the ubatch bench on 8k context), f16 KV, full 262144 context / 1 slot, TEMPERATURE 0.7 / TOP_P 0.95, reasoning on (deepseek, budget -1), --no-mmap. Two-file payload (~14.4 GiB) in /home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP/. Full runbook: serverAppConfigs-docs/LLM/eemin-rocmi4-mtp.md.

Kairic Edge llama.cpp (Dankclaw, RETIRED experiment 2026-08-22)

The Kairic Edge llama-server serves jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (pinned tag kairic-edge-qwen38-27b-v1). It was deployed and reverted to v3 the same day (operator decision). The runtime routes qualified prompt and verification shapes through the RDNA 3.5 native IU4 matrix instruction.

Release gap — CK config.h: the pinned CK commit ships only config.h.in. The release script and guide never generate third_party/composable_kernel/include/ck/config.h. Generate it via a CK CMake configure and copy it in before building (full runbook in serverAppConfigs-docs/LLM/kairic-edge.md). The file is untracked in CK git. A git clean removes it. Local rebuilds are byte-identical (d7625a0c...).

Target greedy argmax fast path: the pinned runner sets LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1. The fast path rejects non-greedy requests (400) and the card's own chat sampling is non-greedy, so the production attempt ran it at 0. It still rejects reasoning budgets unless compatibility mode is used. Do not re-deploy Kairic without re-testing chat first.

Rollback target: the v3 model files remain in /home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/ and the v3 binary in /home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server. Restored to v3 on 2026-08-22 (commit c60156f).

DFlash2 llama.cpp (Dankclaw, retired 2026-08-21)

A DFlash2 Vulkan deployment briefly replaced CIRU on 2026-08-21, then was reverted the same day for speed. Its serve script, env, and the build worktree are archived in .AGENTTRASH/ and documented in serverAppConfigs-docs/LLM/dflash2-large-vulkan.md.

FastMTP patched llama.cpp (Dankserver, retired 2026-08-18)

Historical: the Qwen3.8-27B IQ4_XS FastMTP service was retired on 2026-08-18 (files in .AGENTTRASH/). The build notes are kept for reference. The HauhauCS FastMTP draft sidecar required a llama.cpp build from commit 4df29be4f4c3673f428170fda944a5b19f743bb8 plus the HauhauCS-FastMTP-llama.cpp.patch (from the HF repo). The build lives at /home/dank/llama.cpp-fastmtp/build on Dankserver. Loading the draft with a stock binary fails with expected 5120, 248320, got 5120, 32768. Do not rebuild from another commit without re-applying the patch.

REQUIRED build flags: -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON. Without GGML_CUDA_FA_ALL_QUANTS the q8_0 KV cache + flash attention falls back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README for the base recipe.

Env var naming rule — HIP runtime collision

The ROCm HIP runtime reads the bare environment variables TEMP, HOST, and GPU_MAX_HEAP_SIZE. The serve scripts export the env file with set -a. A tunable named TEMP leaks into HIP. On 2026-08-13 TEMP=1.0 made HIP use the path 1.0 for its JIT kernel cache, then segfaulted every model load. Commit 227ce01 fixed it by renaming the sampling variable to TEMPERATURE.

Never name a tunable TEMP, HOST, or GPU_MAX_HEAP_SIZE. Check for collisions with this command:

strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"

Crash-loop diagnostic procedure

When a llama service segfaults during model load, follow these steps.

graph TD
    A[Service SEGVs at load] --> B[Read journal + kernel segfault line]
    B --> C{Segfault in libamdhip64?}
    C -- yes --> D[Bisect script vs raw binary]
    C -- no --> E[Check config, memory, model files]
    D --> F[Run gdb for backtrace]
    F --> G[Find root cause]

1. Read the service journal with journalctl -u <service> -n 100. Then read the kernel segfault line with journalctl -k. A segfault at address 8 in libamdhip64 means a NULL unref in the HIP runtime.
2. Run the serve script directly. Then run the raw binary with the same flags. A different outcome isolates the env file as the cause.
3. Run gdb for the backtrace with this command:

   gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-small.sh
   

   llama-server buffers its timed stdout. On SIGSEGV the log tail is lost. Trust gdb, not the log.
4. Kernel spam about amdgpu: SVM mapping failed is a symptom, not the root cause. Investigate other causes before assuming memory pressure.

Operational quirks

• Both hosts allow passwordless sudo for the deploy user ((ALL) NOPASSWD: ALL, verified 2026-08-22). Use plain sudo in committed scripts. Do not use echo password | sudo -S in committed scripts. The FRANK_SUDO_PW env var covers headless runs on a host without that.
• Never run pkill -f with a pattern that appears in your own command line. It kills your ssh session. Use pkill -x <exact-name>.
• Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (--no-mmap, MoE experts on CPU) in a few minutes cold. Dankclaw loads the ~30 GiB Q8_0 AGENT payload (main + MTP draft + mmproj) a few minutes cold with a fresh KV cache; ~18 s with a warm page cache. Health returns 503 while loading. Do not kill the process. deploy.sh waits up to 20 minutes on both targets (see LLM_WAIT).
• A client abort and a server release look different in the journal. A client abort logs W srv stop: cancel task, id_task = N. A natural stop (EOS or limit) logs only slot release with truncated = 0. The server never preempts a running generation. When all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in serverAppConfigs-docs/LLM/deployment.md.
• ComfyUI on Dankclaw runs CPU-only (--cpu). ROCm wheels do not support gfx1151. Do not use CUDA images.
• SearXNG outbound traffic must stay on the Gluetun tunnel. Do not move its ports off the gluetun service.
• Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical. See serverAppConfigs-docs/LLM/dankserver-vram-growth.md and serverAppConfigs-docs/LLM/qwen3.8-spec-matrix.md. The FastFlowLM NPU endpoint is retired. See serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md.

Dankclaw memory model

The GPU budget on Dankclaw is the rocm-smi figure: 64 GiB VRAM reported by rocm-smi (SVM-backed). The docs' older 96 GiB figure was the addressable SVM pool. The GPU node reports local_mem_size 0, so all allocations are GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use free or dmidecode for the GPU budget. They show the CPU-side carve-out. This is intentional. Do not change it.

Why

Direct edits on the remote drift from version control. Future agents will not know what changed. Git is the single source of truth. The two workloads share two hosts, one Caddy, and one Gitea. One repo keeps them in one history.

User request → Fetch from remote → Commit here → Edit → Test → Commit → Push → Pull on remote → Dispatch changed service(s) → Health

Summary

Context	Action
Editing a compose/Caddyfile/serve script	Work through git — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote directly.
Config exists only on remote	Pull it into the repo first — commit the fetched version (redacted) before making changes.
Unstaged local changes	Commit or discard — check git status before starting work. Ask the user if unsure.
Deploying a committed change	Run scripts/deploy.sh <target> — it pushes, pulls with fast-forward-only, plans changed paths, dispatches selected services, and checks their health. See serverAppConfigs-docs/LLM/deployment.md.
Benchmarking	Stop the service first, restart, verify health. See the Benchmarking Policy above.
Deleting a file	Move it to trash — .AGENTTRASH/ (in-repo) or ~/.AGENTTRASH/YYYYMMDD-topic. Never rm.
Tests exist	Run them — verify tests pass after every change. Do not commit failing tests.
Real secrets	Never commit — use host .env files. .env.example holds placeholders only. FRANK_SUDO_PW covers headless sudo.