serverAppConfigs/AGENTS.md

# AGENTS.md — serverAppConfigs Agent Rules

## Overview

This repository is the single source of truth for the two homelab servers. It
tracks both the Podman container stacks and the LLM inference fleet. One repo,
one checkout per host, one deploy script.

- **Dankserver** (192.168.0.235) — media/library stack (RomM, Copyparty,
  Calibre-Web, the *arr stack, Jellyfin) as bare containers, plus the
  `frank-small` LLM service.
- **Dankclaw** (192.168.0.143) — AI/search/development/observability stack
  (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus,
  Loki, Alloy, Pi-hole) plus the `frank-large` LLM service. Caddy on
  Dankclaw reverse-proxies both hosts.

**2026-08-22 merge:** the former `vllmScripts` repository (LLM fleet scripts,
units, env tunables, and runbook) is merged into this repo. Its full git
history is preserved (two-parent import commit under the retired `llm/`
prefix, reorganized in follow-up commits). The retired target checkouts
(`/home/dank/vLLM` on Dankserver, `/home/dank/vllmScripts` on Dankclaw) are
archived. All services now run from `/home/dank/serverAppConfigs` on each
host.

### LLM fleet topology (2026-08-30)

| Target | Runtime | Service | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | FreeToken v0.1.2 | frank-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 |
| Dankclaw | kingjones30/ROCmFPX llama.cpp (card-pinned fork) | frank-large | kingjones777 Qwen3.8-Flash-Next-ROCmFP4-STRIX (113.47 GiB single GGUF, Q4_0_ROCMFP4_STRIX, PLE streams via mmap), 401408 ctx / 2 slots, fa on, no spec decode, reasoning on | 7770 |
| Dankclaw | FastFlowLM (FLM) v1.0.1 (`flm serve`, dpkg `fastflowlm`) on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio towers); whole local registry served on one port, per-request routing by exact FLM tag; `iommu=pt` boot arg mandatory (SVA bind needs enabled IOMMU) | 7771 |

History: the 2026-08-17 redo retired the pipeline-parallel split and the
FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
Dankclaw. On 2026-08-21 a DFlash2 Vulkan deployment briefly replaced CIRU,
then was reverted the same day for speed. On 2026-08-22 the service briefly
swapped to the Kairic Edge IU4 runtime, then was reverted to v3 the same day
(operator decision). On 2026-08-25 the service switched to the kingjones
Qwen3.8-27B-Uncensored-Q6_0-ROCmFPX_AGENT model (standalone MTP draft,
262144 context, reasoning on). On 2026-09-01 both services were swapped:
Dankserver switched from ik_llama.cpp Ornith CRACK Q8_0 to stock llama.cpp
Tiel-Coder-35B (qwen35moe hybrid MoE), and Dankclaw switched from the Q6_0
AGENT model to the Flash-Next 56B MoE (qwen4exp, PR #27742 merged into
charlie's ROCmFPX fork, --fit off, 128k context). On 2026-09-02
Dankserver moved to the FreeToken runtime serving the official
Qwen3.6-35B-A3B-FP8. On 2026-08-30 both services swapped again:
Dankclaw to the kingjones STRIX model on the card's pinned
kingjones30/ROCmFPX runtime, Dankserver to the Huihui abliterated
FP8 checkpoint on FreeToken. On 2026-08-30 the NPU endpoint returned as
`frank-npu` — FLM v1.0.1 on XDNA2 serving sight+sound (see the
Runtime & Troubleshooting section and its runbook). Runbooks:
[serverAppConfigs-docs/LLM/index.md](serverAppConfigs-docs/LLM/index.md).
Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`.

## Repository Layout

- `dankserver/` — compose.yaml + Caddyfile + .env.example for the media stack
- `dankclaw/` — compose.yaml + Caddyfile + .env.example for the AI stack
- `shared/` — homepage, Prometheus, Loki, Alloy, SearXNG configs
- `scripts/` — `deploy.sh`, `deploy-plan.py`, and `deploy-remote.sh`
  (service-scoped deployer), LLM serve scripts, systemd units, env tunables,
  and bench harnesses
- `old/` — retired PP/NPU-era LLM scripts (reference only)
- `serverAppConfigs-docs/` — knowledge base (Obsidian vault). `serverAppConfigs-docs/LLM/` is the LLM fleet
- `tests/` — bats test suite

## Documentation Authoring

The `serverAppConfigs-docs/` vault is the project knowledge base (Obsidian, wikilinks) and is an
OKF (Open Knowledge Format) knowledge bundle. Follow the OKF spec for every doc
you create or edit.

- **Canonical spec:** `~/.pi/agent/docs/references/okf-spec.md` — the OKF v0.1
  spec in the **system-config knowledge base** at `~/.pi/agent/docs`. The repo's
  own `serverAppConfigs-docs/references/okf-spec.md` is deprecated/removed; use the system one. The
  spec index and maintenance guide are
  `~/.pi/agent/docs/references/index.md` and
  `~/.pi/agent/docs/references/okf-maintenance.md`.
- **Read before you touch:** open the relevant `index.md` first
  (`serverAppConfigs-docs/index.md` → `serverAppConfigs-docs/LLM/index.md`). Read the whole target file before
  editing it.
- **One concept per file**, with YAML frontmatter that has a non-empty `type:`
  field (`reference` / `guide` / `configuration` / `research`) plus `description`,
  `tags`, and `timestamp`. Split broad topics; keep pages ~1–500 lines.
- **Progressive disclosure:** an `index.md` at each level links its children.
  Every `serverAppConfigs-docs/**/*.md` (except `index.md`) must be linked from
  `serverAppConfigs-docs/index.md` with `[[Title]]` — the docs-completeness test checks this.
- **Cross-link** siblings as `[[Title]]` wikilinks; cite external claims under a
  `# Citations` section.
- **Always update docs** when a task changes behavior, then run
  `./tests/run.sh` (the docs-completeness seam is in there) before commit.

## Git Workflow

All edits flow through git. Do not edit files on the remote machine directly.

### Protocol

1. **Source the config from the target machine** — fetch the live
   compose/Caddyfile/serve script from the server where it runs
2. **Place it in this repo** — commit the fetched version (redact secrets
   into `.env`)
3. **Edit here** — make all changes locally in the working tree
4. **Run the tests** — `./tests/run.sh` must pass
5. **Commit and push** — push to origin (Gitea on Dankclaw)
6. **Pull on the target** — `git pull` on the remote machine
7. **Deploy** — `scripts/deploy.sh` pulls, plans, and dispatches only the
   changed service(s), then checks the selected service health

## Deployment

Use `scripts/deploy.sh` to deploy committed changes to a target. The planner
maps changed paths to individual services; the remote runner restarts only the
selected service and records its successful revision outside the checkout.

```
./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankserver
./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-small
./scripts/deploy.sh dankclaw --dry-run
```

A documentation-only change is a no-op. The LLM services are selected only by
their own unit/env/serve files or the shared chat template, so unrelated
container changes cannot interrupt inference. `frank-npu` is relink-only
when explicitly selected.

Set `FRANK_SUDO_PW` for headless runs on a host without passwordless sudo
(remote sudo reads it from stdin via `sudo -S`). The value is never stored
in this repo.

```mermaid
graph LR
    A[Edit locally] --> B[Tests pass]
    B --> C[Commit]
    C --> D[Push to origin]
    D --> E[deploy.sh computes service plan]
    E --> F[Target pulls with ff-only]
    F --> G[deploy-remote.sh dispatches selected service]
    G --> H[Per-service health check]
```

## Boot Persistence & Self-Healing (MANDATORY)

Every Podman container governed by this project that is successfully deployed
and proven working MUST have a self-healing, boot-persistent Quadlet unit. A
container that runs only from `compose.yaml` (or a manual `podman run`) dies
silently at reboot: rootless Podman does not auto-start bare containers, and
`restart: unless-stopped` does not survive the explicit stop that shutdown
performs. Do not report such a service as "up" without its unit.

Each container's unit must provide:

- **Boot persistence** — `[Install] WantedBy=default.target`, installed to
  `~/.config/containers/systemd/<name>.container` (generating
  `<name>.service`) and enabled in the user manager
  (`systemctl --user enable <name>.service`). Linger must be on for the deploy
  user (`loginctl show-user dank -p Linger` → `Linger=yes`) so user units start
  at boot with no login session.
- **Self-healing** — `[Service] Restart=always` (or `Restart=on-failure`) with
  a short `RestartSec`, so a crash or OOM brings the container back without
  operator action. Add a quadlet health check where the service has one.
- **Repo provenance** — the `.container` file lives in this repo
  (`dankclaw/<svc>/quadlet/` or `dankserver/<svc>/quadlet/`), is installed by
  the service-scoped deployer, and is covered by the quadlet test seam. The
  host-side unit is generated state; the repo file is the source of truth.

Verification before a service is declared deployed:
`systemctl --user is-enabled <name>.service` → `enabled`,
`systemctl --user is-active <name>.service` → `active`, and the container
answers its health check. A running container with no unit is a gap: write the
unit, test it, deploy it.

Incident that established this rule (2026-08-26, Dankclaw reboot): Caddy ran
for weeks as a bare compose container and never came back after reboot; the
Pi-hole unit existed but crash-looped on a sysctl conflict (a later-sorted
drop-in reset `net.ipv4.ip_unprivileged_port_start` to 80), taking LAN DNS
down with it.

## Testing

Tests are mandatory before commit. `bats` must be installed on the
development machine.

```
./tests/run.sh
```

Test seams: compose validity, env consistency (no leaked secrets), Caddyfile
validity, quadlet units, docs completeness, script syntax (all of `scripts/`,
including the LLM serve scripts), LLM unit integrity (units point at files in
this repo), LLM secret hygiene, deployment flow, and service-scoped deployment
planning/dispatch. See [serverAppConfigs-docs/Architecture/Testing Policy.md](serverAppConfigs-docs/Architecture/Testing%20Policy.md).

## Secret Rules

- Never commit real secrets. The live Gitea database password and the live
  sudo password are test-enforced as leak checks.
- `.env` files are git-ignored. Commit only `.env.example` with placeholders.
  The LLM env tunables in `scripts/` are tracked by exception (`!scripts/*.env`).
  They hold paths and numbers, never credentials.
- Do not document passwords in this repo, including sudo passwords. SSH keys
  are the auth mechanism. Headless sudo passes the password via the
  `FRANK_SUDO_PW` environment variable. The value never enters the repo.


## External Exposure Policy (MANDATORY)

Every service reverse-proxied through Caddy is either behind the auth chain
(Anubis :8923 → Authentik embedded outpost) or explicitly designated
internal-only — no public DNS name, no vhost. New services default to
internal-only until guarded.

When guarding a new vhost, follow this order — provider first, then route:

1. Create the outpost proxy provider FIRST (`external_host` = the exact vhost
   URL including trailing slash; `internal_host = http://caddy:8017`). If the
   vhost flips before the provider exists, authenticated traffic hits the
   outpost error page instead of the app.
2. Flip the vhost to `reverse_proxy anubis:8923` with
   `header_up X-Real-Ip {remote_host}`, and add the post-auth
   `@<svc> host <vhost>` handler on the `http://:8017` listener routing to the
   real upstream.
3. The Anubis guard set, the Caddy TLS edge, and the outpost provider list
   must stay in lockstep — they change together in one deploy.
4. Service-to-service traffic (e.g., Alloy → Loki push) targets internal
   addresses (`loki:3100`), never the public hostname — non-browser clients
   cannot solve challenges.

Exempt-by-construction categories stay exempt with a documented reason in the
vhost comment: the IdP itself (`auth.` — its login flow must never sit behind a
challenge), git smart-HTTP endpoints (git clients cannot solve JS challenges),
and services with their own complete auth.
## Deletion & Trash Policy (MANDATORY)

Never run `rm` to delete files. The command deletes permanently. Move every
item you delete into trash instead. The trash keeps the deletion recoverable
and reviewable.

1. In-repo files: move the item to `.AGENTTRASH/` beside its directory.
2. Home and other paths: create a dated staging directory
   `~/.AGENTTRASH/YYYYMMDD-topic` and move each item there.
3. List the staged items and their sizes for the user.
4. Wait for explicit user permission before you empty the trash.

Rules:

- `rm`, `rm -rf`, and `git clean -f` are forbidden without explicit user
  permission.
- `git * --no-verify` is forbidden.
- A moved item stays in the trash until the user approves its purge.
- The trash is a temporary hold, not a long-term archive.

## Benchmarking Policy (MANDATORY)

**STOP the production service BEFORE running any benchmark on a target.**
These servers serve real users (about 3 people). Running a bench while the
service is up starves the GPU and corrupts results. Restart the service when
the bench finishes.

ALWAYS bench using llama-bench for llama.cpp or vllm bench for vllm runtimes.
DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE OFFICIAL BENCHES. ASK THE USER BEFORE DEVIATING FROM THIS DIRECTIVE.

1. Stop the service before any benchmark:
   `sudo systemctl stop frank-large` (Dankclaw) or
   `sudo systemctl stop frank-small` (Dankserver).
   Then mask it to prevent the self-healing unit from restarting:
   `sudo systemctl mask frank-large` (Dankclaw).
   **CRITICAL WARNING:** Do NOT use `systemctl disable` on these service units.
   On these hosts, `systemctl disable` doesn't just remove symlinks — it **deletes
   the unit file itself** from `/etc/systemd/system/`. You'd have to manually
   copy the unit file back from this repo, run `daemon-reload`, then enable and
   start. Mask (`systemctl mask`) creates a symlink to `/dev/null` instead, which
   is safe and reversible.
2. Run the bench (`llama-bench` for llama.cpp / vllm bench for vLLM).
   Scratch ports are not used for tests or benches on the LLM hosts — a
   test process cannot share the single GPU with the running service anyway
   (the model + KV fill the card), so changes are validated directly against
   the service itself.
3. Restart the service when the bench completes: unmask, then start:
   `sudo systemctl unmask frank-large && sudo systemctl start frank-large` (Dankclaw).
4. Verify health before reporting:
   `curl http://192.168.0.235:7770/health` or
   `curl http://192.168.0.143:7770/health`.

## Pi Model Config (`~/.pi/agent/models.json`)

The Pi agent reaches the fleet through `/home/dank/.pi/agent/models.json`.
This file lives outside the repository. Each provider is one host endpoint.
Each model `id` is the served alias. The two aliases are
`frank-small` (Dankserver `:7770`) and `frank-large`
(Dankclaw `:7770`).

The model `id` values are stable. The file
`/home/dank/.pi/agent/settings.json` references them. Two resolution rules
apply (verified in `@earendil-works/pi-coding-agent` `model-resolver.js`):

- The default model resolves as a `defaultProvider` + `defaultModel` pair.
  The `defaultProvider` must name the provider that serves the `defaultModel`.
- A subagent override uses a bare model `id`. A bare `id` resolves only when
  it is unique across all providers.

When you rename a service or move a model to a new host, update
`models.json`. Update the provider key, the model `id`, `contextWindow`, and
`input`. If the default model moved hosts, also update `defaultProvider` in
`settings.json`. Set `contextWindow` to the per-slot context (both hosts
`262144`). The full schema is in the agent knowledge base page
`~/.pi/agent/docs/pi/models.md`.

## Runtime & Troubleshooting

### frank-npu — FLM NPU sight+sound endpoint (Dankclaw, ACTIVE)

The third inference endpoint on Dankclaw. The `flm serve` runtime (dpkg
package `fastflowlm`, v1.0.1, binary `/usr/bin/flm`) serves the **whole local
model registry** (36 tags at first deploy) on a single port — 7771 — and
routes chat completions per request by the exact FLM registry tag in the
`model` field. Unknown tags do NOT 404: `/v1/chat/completions` answers HTTP 200
with `{"error":"Failed to load <tag> model!"}` and no choices (status-only
clients silently get nothing), so clients must send exact registry tags, never
service aliases. Primary
model = Gemma4 E4B IT NPU2 pack (vision + audio towers); the uncensored
candidate (`qwythos:9b`) is text-only and fails the modality requirement, so
this endpoint trades "uncensored" for "sight + sound". The NPU is a
single-lane device — generations serialize FIFO; treat this as a
background/triage endpoint, not an interactive multi-user chat.

Hard requirements verified at first deploy (2026-08-30):

- **IOMMU must be enabled.** With `iommu=off` on the kernel cmdline every
  open of `/dev/accel/accel0` fails with `err=-19`: the XDNA2 driver's SVA
  bind needs PASID/SVA, which requires enabled IOMMU hardware. Fix applied:
  `iommu=pt` in `GRUB_CMDLINE_LINUX_DEFAULT` + reboot (passthrough keeps the
  iGPU default domain identity-mapped — zero DMA-translation overhead for
  frank-large — while enabling IOMMU + PASID; PASID domains are
  paged regardless of the default-domain type). Never set `iommu=off` on
  this host again; it silently kills the NPU.
- Health probe = `GET /v1/version` (FLM has no `/health` route).
- Unit: `User=dank`, `LimitMEMLOCK=infinity` (FLM requires unlimited
  memlock), `Restart=on-failure`, `RestartSec=10`. Deploy via
  `scripts/deploy.sh dankclaw --service frank-npu`.
- Verified live at first deploy: image probe (red PNG as `image_url`
  data-URI → correct color answer, vision tower consumed the pixels) and an
  audio control-pair probe (tone WAV via `input_audio` consumed ~22
  audio-tower tokens vs a text-only control).

Runbook: [serverAppConfigs-docs/LLM/frank-npu.md](serverAppConfigs-docs/LLM/frank-npu.md).
### kingjones Qwen3.8-Flash-Next STRIX on kingjones30/ROCmFPX (Dankclaw, ACTIVE)

The large service serves
`kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` (public, 113.47
GiB single file, `Q4_0_ROCMFP4_STRIX` = file_type 105) from
`/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/` with
the card's **pinned runtime**: the `kingjones30/ROCmFPX` fork
(charlie12345/ROCmFPX + qwen4exp-on-rocmfpx-d3ca537 patch — ROCmFP4
types and qwen4exp arch in one tree), built at
`/home/dank/GitPackages/kingjones-ROCmFPX/build` with
`-DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON
-DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_WEBUI=OFF` (WebUI
provisioning needs npm/HF-bucket and aborts the link at 98% without
the OFF flag). Args: our validated profile — 401408 context / 2 slots,
b 4096 / ub 512 (ub 512 matches the card's re-measurement), t 16 / tb
16, f16 KV, `--fit off`, `--flash-attn on` (works UNPATCHED on this
model — the QSA F16 ggml_fill assert that needed
`scripts/patches/qwen4exp-qsa-f32-mask.patch` is specific to the
agentionai FAST layer set; if a future build aborts with
`GGML_ASSERT(a-type == GGML_TYPE_F32)`, port that fix), reasoning on
(Sharp template, deepseek format), no speculative decoding (card ships
no spec flags; Flash-Next MTP heads do not load on this loader family).
Two hard card constraints: **mmap must stay on** (`--no-mmap` forces
the 50.66 GiB Q8_0 PLE table into anonymous memory → silent cgroup
kill, only dmesg evidence; llama.cpp's "use_mmap=1" log line is
correct here — mmap pages ARE page cache) and **never pin the PLE
tensor with `-ot`** (23 → 13.4 t/s). The 51.2B-param PLE table never
enters GPU memory (~20 bytes/token-file per 500 decoded). On-host file
sha256 verified == HF LFS oid `c6770d74...49fa`. Env
`scripts/frank-large.env`, serve `scripts/serve-frank-large.sh`.
Runbook:
[serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md](serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md).

**Serving status (2026-08-30):** swapped from the agentionai FAST
deployment. Validated live before the systemd handoff: clean load,
health 200, reasoning completions, and an operator session's ~111k
token prefill at 213–232 t/s (card rung: 196 at 131k). Note llama.cpp
clamps n_ctx_train 262144 → per-slot 131072 with parallel 2.

### agentionai ROCmFP4-FAST imatrix llama.cpp (Dankclaw, prior, 2026-08-29 to 2026-08-30)

Historical — rollback target. `agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF`
(87.06 GiB, Q4_0_ROCMFP4_FAST imatrix, v2/ joined-PLE layout only — the
root `-ple16` per-head split aborts this loader family) on the charlie
`build-qwen38` tree (`/home/dank/GitPackages/ROCmFPX`, PR #27742
merged) at 32768/1, ub 4096. That tree REQUIRES
`scripts/patches/qwen4exp-qsa-f32-mask.patch` for `--flash-attn on`
(F16-kq_mask ggml_fill abort otherwise). Flash-Next MTP heads do not
load on any current loader (require full per-layer
`blk.<il>.hc_attn_norm.weight`; heads carry only `blk.48` — verified
3x). Binary and payload retained. Runbook:
[serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md](serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md).

### Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on FreeToken (Dankserver, ACTIVE)

The small service serves `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8`
(public, ~35 GB, 26 safetensors shards) from
`/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8/` with
FreeToken v0.1.2 (`/home/dank/freetoken/.venv`, `ft serve`). Native
modelopt block-fp8 (e4m3, dynamic, 128×128 weight_block_size,
un-fused per-expert layout) — same format and loader path as the
official Qwen3.6-35B-A3B-FP8 it replaced on 2026-08-30, so the entire
memory/pool plan carries: experts host-RAM with 20% LRU GPU residency,
paged KV (only the 10 full-attention layers page; 30 GDN layers use
constant-size mamba slots), ~3.4 GiB VRAM headroom at boot, and the
validated `--memory-num-gpu-blocks-experts 2048
--memory-num-gpu-tokens-kv-cache 453005 --max-num-seqs 4` pool triple
(never grow the expert pool). 262144 context, reasoning on (the
checkpoint ships the froggeric fixed chat template — no override).
FreeToken's API REQUIRES an explicit `"model"` field in chat requests
(422 without it). Env `scripts/frank-small.env`, serve
`scripts/serve-frank-small.sh`. Runbook:
[serverAppConfigs-docs/LLM/freetoken-frank-small.md](serverAppConfigs-docs/LLM/freetoken-frank-small.md).

**Serving status (2026-08-30):** swapped the model within the FreeToken
runtime (official FP8 → Huihui abliterated FP8). ~40 tok/s verified on
a 282-token completion post-deploy. Prior: official Qwen FP8
(2026-09-02, runtime swap), Tiel-Coder-35B on llama.cpp (2026-09-01),
ik_llama.cpp Ornith CRACK (retired 2026-09-02).

### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, prior v3, 2026-08-18 to 2026-08-25)

The v4 runtime serves `cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (`eemin` namespace,
public) from `/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server`
(charlie12345/ROCmFPX commit `c49ebdbd` built with `-DGGML_HIP_ROCMI4_W4A4=ON`
against system ROCm 7.2.4 / amdclang 22). The model is `Q4_0_ROCMI4` with an
embedded NextN/MTP head (no separate draft file); the gfx1151 native IU4 WMMA
instruction batch-verifies proposals. Profile: strict Qwen MTP
(`--spec-mtp-strict-qwen`, n_max 16, p_min 0.60, backend sampling, all layers
on ROCm0), `-t 16 -tb 32`, batch 512 / ubatch 256 (ubatch swappable via the
ubatch bench on 8k context), f16 KV, full 262144 context / 1 slot, TEMPERATURE
0.7 / TOP_P 0.95, reasoning on (deepseek, budget -1), `--no-mmap`. Two-file
payload (~14.4 GiB) in `/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP/`. Full
runbook:
[serverAppConfigs-docs/LLM/eemin-rocmi4-mtp.md](serverAppConfigs-docs/LLM/eemin-rocmi4-mtp.md).


### Kairic Edge llama.cpp (Dankclaw, RETIRED experiment 2026-08-22)

The Kairic Edge llama-server serves `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge`
(pinned tag `kairic-edge-qwen38-27b-v1`). It was deployed and reverted to v3
the same day (operator decision). The runtime routes qualified prompt and
verification shapes through the RDNA 3.5 native IU4 matrix instruction.

**Release gap — CK `config.h`:** the pinned CK commit ships only
`config.h.in`. The release script and guide never generate
`third_party/composable_kernel/include/ck/config.h`. Generate it via a CK
CMake configure and copy it in before building (full runbook in
[serverAppConfigs-docs/LLM/kairic-edge.md](serverAppConfigs-docs/LLM/kairic-edge.md)). The file is untracked
in CK git. A `git clean` removes it. Local rebuilds are byte-identical
(`d7625a0c...`).

**Target greedy argmax fast path:** the pinned runner sets
`LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1`. The fast path rejects non-greedy
requests (400) and the card's own chat sampling is non-greedy, so the
production attempt ran it at 0. It still rejects reasoning budgets unless
compatibility mode is used. Do not re-deploy Kairic without re-testing chat
first.

**Rollback target:** the v3 model files remain in
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/` and the v3 binary in
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
Restored to v3 on 2026-08-22 (commit `c60156f`).

### DFlash2 llama.cpp (Dankclaw, retired 2026-08-21)

A DFlash2 Vulkan deployment briefly replaced CIRU on 2026-08-21, then was
reverted the same day for speed. Its serve script, env, and the build
worktree are archived in `.AGENTTRASH/` and documented in
[serverAppConfigs-docs/LLM/dflash2-large-vulkan.md](serverAppConfigs-docs/LLM/dflash2-large-vulkan.md).

### FastMTP patched llama.cpp (Dankserver, retired 2026-08-18)

Historical: the Qwen3.8-27B IQ4_XS FastMTP service was retired on 2026-08-18
(files in `.AGENTTRASH/`). The build notes are kept for reference. The
HauhauCS FastMTP draft sidecar required a llama.cpp build from commit
`4df29be4f4c3673f428170fda944a5b19f743bb8` plus the
`HauhauCS-FastMTP-llama.cpp.patch` (from the HF repo). The build lives at
`/home/dank/llama.cpp-fastmtp/build` on Dankserver. Loading the draft with a
stock binary fails with `expected 5120, 248320, got 5120, 32768`. Do not
rebuild from another commit without re-applying the patch.

**REQUIRED build flags:** `-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON`.
Without `GGML_CUDA_FA_ALL_QUANTS` the q8_0 KV cache + flash attention falls
back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s
vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README
for the base recipe.

### Env var naming rule — HIP runtime collision

The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`,
and `GPU_MAX_HEAP_SIZE`. The serve scripts export the env file with
`set -a`. A tunable named `TEMP` leaks into HIP. On 2026-08-13 `TEMP=1.0`
made HIP use the path `1.0` for its JIT kernel cache, then segfaulted every
model load. Commit `227ce01` fixed it by renaming the sampling variable to
`TEMPERATURE`.

Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. Check for
collisions with this command:

```
strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"
```

### Crash-loop diagnostic procedure

When a llama service segfaults during model load, follow these steps.

```mermaid
graph TD
    A[Service SEGVs at load] --> B[Read journal + kernel segfault line]
    B --> C{Segfault in libamdhip64?}
    C -- yes --> D[Bisect script vs raw binary]
    C -- no --> E[Check config, memory, model files]
    D --> F[Run gdb for backtrace]
    F --> G[Find root cause]
```

1. Read the service journal with `journalctl -u <service> -n 100`. Then read
   the kernel segfault line with `journalctl -k`. A segfault at address 8 in
   `libamdhip64` means a NULL unref in the HIP runtime.
2. Run the serve script directly. Then run the raw binary with the same
   flags. A different outcome isolates the env file as the cause.
3. Run gdb for the backtrace with this command:
   ```
   gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-large.sh
   ```
   `llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost.
   Trust gdb, not the log.
4. Kernel spam about `amdgpu: SVM mapping failed` is a symptom, not the root
   cause. Investigate other causes before assuming memory pressure.

### Operational quirks

- Both hosts allow passwordless sudo for the deploy user (`(ALL) NOPASSWD:
  ALL`, verified 2026-08-22). Use plain `sudo` in committed scripts. Do not
  use `echo password | sudo -S` in committed scripts. The `FRANK_SUDO_PW`
  env var covers headless runs on a host without that.
- Never run `pkill -f` with a pattern that appears in your own command line.
  It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM
  (`--no-mmap`, MoE experts on CPU) in a few minutes cold. Dankclaw loads the
  ~30 GiB Q8_0 AGENT payload (main + MTP draft + mmproj) a few minutes cold
  with a fresh KV cache; ~18 s with a warm page cache. Health returns 503 while loading. Do not kill the
  process. `deploy.sh` waits up to 20 minutes on both targets (see
  `LLM_WAIT`).
- A client abort and a server release look different in the journal. A
  client abort logs `W srv stop: cancel task, id_task = N`. A natural stop
  (EOS or limit) logs only `slot release` with `truncated = 0`. The server
  never preempts a running generation. When all slots are busy, new tasks
  defer into a queue. See the 2026-08-13 incident in
  [serverAppConfigs-docs/LLM/deployment.md](serverAppConfigs-docs/LLM/deployment.md).
- ComfyUI on Dankclaw runs CPU-only (`--cpu`). ROCm wheels do not support
  gfx1151. Do not use CUDA images.
- SearXNG outbound traffic must stay on the Gluetun tunnel. Do not move
  its ports off the gluetun service.
- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash
  premature-EOS bug are historical. See
  [serverAppConfigs-docs/LLM/dankserver-vram-growth.md](serverAppConfigs-docs/LLM/dankserver-vram-growth.md)
  and [serverAppConfigs-docs/LLM/qwen3.8-spec-matrix.md](serverAppConfigs-docs/LLM/qwen3.8-spec-matrix.md).
  The FastFlowLM NPU endpoint is retired. See
  [serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md](serverAppConfigs-docs/LLM/FastFlowLM%20NPU%20Inference.md).

### Dankclaw memory model

The GPU budget on Dankclaw is the rocm-smi figure: 64 GiB VRAM reported by
rocm-smi (SVM-backed). The docs' older 96 GiB figure was the addressable
SVM pool. The GPU node reports `local_mem_size 0`, so all allocations are
GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use `free` or
`dmidecode` for the GPU budget. They show the CPU-side carve-out. This is
intentional. Do not change it.

## Why

Direct edits on the remote drift from version control. Future agents will not
know what changed. Git is the single source of truth. The two workloads share
two hosts, one Caddy, and one Gitea. One repo keeps them in one history.

> User request → Fetch from remote → Commit here → Edit → Test → Commit → Push → Pull on remote → Dispatch changed service(s) → Health

## Summary

| Context | Action |
|---------|--------|
| Editing a compose/Caddyfile/serve script | **Work through git** — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote directly. |
| Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before making changes. |
| Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. |
| Deploying a committed change | **Run `scripts/deploy.sh <target>`** — it pushes, pulls with fast-forward-only, plans changed paths, dispatches selected services, and checks their health. See [serverAppConfigs-docs/LLM/deployment.md](serverAppConfigs-docs/LLM/deployment.md). | | Container without a Quadlet unit | **Give it one** — self-healing, boot-persistent `.container` unit via git (see Boot Persistence & Self-Healing). |
| Benchmarking | **Stop the service first**, restart, verify health. See the Benchmarking Policy above. |
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
| Real secrets | **Never commit** — use host `.env` files. `.env.example` holds placeholders only. `FRANK_SUDO_PW` covers headless sudo. |