From f2277863aed0f4ce1fd1652b5079ebf9a1cb1291 Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 16 Sep 2026 00:20:58 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20rewrite=20AGENTS.md=20=E2=80=94=20dedup?= =?UTF-8?q?e=20fleet=20history=20into=20lessons=20+=20runbook=20index?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - 734 -> 412 lines; active topology table replaces the swap-history log - retired-runtime detail collapsed into a runbook pointer table (serverAppConfigs-docs/LLM/ holds the full runbooks) - halogen section condensed to invariants that survive config changes (pool/reserve matched pair, free-lies rule, client contract) - new Key Lessons Learned section: ROCm/layer-split, HIP env collisions, mmap/PLE constraints, crash diagnosis, swap ordering, podman/quadlet gotchas, ops guards - fixed the merged/broken row in the Summary table --- AGENTS.md | 904 ++++++++++++++++++------------------------------------ 1 file changed, 291 insertions(+), 613 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 30114b5..7b23194 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,132 +2,59 @@ ## Overview -This repository is the single source of truth for the two homelab servers -and the FrankBook-Pro Mac mini. It tracks the Podman container stacks and -the LLM inference fleet. One repo, one checkout per host, one deploy script. +This repository is the single source of truth for the two homelab servers and +the FrankBook-Pro Mac mini. It tracks the Podman container stacks and the LLM +inference fleet. One repo, one checkout per host, one deploy script. All +services run from `/home/dank/serverAppConfigs` on each host (the 2026-08-22 +merge absorbed the former `vllmScripts` repo, history preserved). - **Dankserver** (192.168.0.235) — media/library stack (RomM, Copyparty, - Calibre-Web, Jellyfin, Plex) as bare containers, plus the `frank-medium` - LLM service. -- **Dankclaw** (192.168.0.143) — AI/search/development/observability stack - (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, - Prometheus, Loki, Alloy, Pi-hole) plus the `frank-large` LLM service. - Caddy on Dankclaw reverse-proxies both hosts. + Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service. +- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG + via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, + Alloy, Pi-hole) plus the `frank-large` LLM service. Caddy on Dankclaw + reverse-proxies both hosts. - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack - (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets - inside its podman machine; every container rides the Gluetun WireGuard - netns, so all *arr and qBittorrent egress is VPN-tunneled. Deployed via - the `frankbook-pro` handler. + (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside + its podman machine; all egress rides the Gluetun WireGuard netns. Deployed + via the `frankbook-pro` handler. -**2026-08-22 merge:** the former `vllmScripts` repository (LLM fleet scripts, -units, env tunables, and runbook) is merged into this repo. Its full git -history is preserved (two-parent import commit under the retired `llm/` -prefix, reorganized in follow-up commits). The retired target checkouts -(`/home/dank/vLLM` on Dankserver, `/home/dank/vllmScripts` on Dankclaw) are -archived. All services now run from `/home/dank/serverAppConfigs` on each -host. +## Fleet topology (ACTIVE) -### LLM fleet topology (2026-09-15) - -| Target | Runtime | Service | Model | Port | +| Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | charlie12345/ROCmFPX llama.cpp | frank-medium | kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF (Q8_0_ROCMFPX_AGENT ftype 115, 26.28 GiB, 8.39 bpw), 2x Radeon AI PRO R9700 layer split, 131072 ctx / 1 slot, f16 KV, MTP draft head on ROCm1, reasoning on | 7770 | -|Dankclaw|halogen-flash-server 0.5.8 (rootless Quadlet container)|frank-large|peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 3 slots, pool 401408 / max_tok 32768 / reserve 15, vision tower on, greedy for silent clients)|7770| -| Dankclaw | *(retired 2026-09-11)* FastFlowLM v1.0.1 on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio). Retired with the `amd_iommu=off` decision: the driver's SVA/PASID bind needs an enabled IOMMU | 7771 | +| Dankserver | `frank-medium` | charlie12345/ROCmFPX llama.cpp (system unit) | kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF, 2x Radeon AI PRO R9700 layer split, 131072 ctx / 1 slot, MTP draft on ROCm1 | 7770 | +| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai/halogen-qwen3.8-flash-next w4b .hgn, 262144 ctx / 3 slots, vision tower on | 7770 | -History: the 2026-08-17 redo retired the pipeline-parallel split and the -FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on -Dankclaw. On 2026-08-21 a DFlash2 Vulkan deployment briefly replaced CIRU, -then was reverted the same day for speed. On 2026-08-22 the service briefly -swapped to the Kairic Edge IU4 runtime, then was reverted to v3 the same day -(operator decision). On 2026-08-25 the service switched to the kingjones -Qwen3.8-27B-Uncensored-Q6_0-ROCmFPX_AGENT model (standalone MTP draft, -262144 context, reasoning on). On 2026-09-01 both services were swapped: -Dankserver switched from ik_llama.cpp Ornith CRACK Q8_0 to stock llama.cpp -Tiel-Coder-35B (qwen35moe hybrid MoE), and Dankclaw switched from the Q6_0 -AGENT model to the Flash-Next 56B MoE (qwen4exp, PR #27742 merged into -charlie's ROCmFPX fork, --fit off, 128k context). On 2026-09-02 -Dankserver moved to the FreeToken runtime serving the official -Qwen3.6-35B-A3B-FP8. On 2026-08-30 both services swapped again: -Dankclaw to the kingjones STRIX model on the card's pinned -kingjones30/ROCmFPX runtime, Dankserver to the Huihui abliterated -FP8 checkpoint on FreeToken. On 2026-08-30 the NPU endpoint returned as -`frank-npu` — FLM v1.0.1 on XDNA2 serving sight+sound. On 2026-09-11 -Dankclaw's large service swapped runtime *and* model: the kingjones STRIX -llama.cpp deployment gave way to halogen-flash-server 0.5.8 serving -peonist-ai/halogen-qwen3.8-flash-next, now a rootless Quadlet container, and -the kernel cmdline moved from `iommu=pt` to `amd_iommu=off` for 13–16 % of -prefill — which retired `frank-npu` for good (see the Runtime & -Troubleshooting section and its runbook). On 2026-09-15 the RTX 3090 left -Dankserver for two Radeon AI PRO R9700 cards (gfx1201): `frank-small` -(FreeToken/CUDA) retired and `frank-medium` (charlie12345/ROCmFPX llama.cpp, -kingjones Q8_0_ROCMFPX-AGENT) took the :7770 endpoint — layer split, not -tensor, because the RCCL-less build's butterfly allreduce collapses prefill -7x (see the runbook for the measured table). Runbooks: -[serverAppConfigs-docs/LLM/index.md](serverAppConfigs-docs/LLM/index.md). -Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`. +Retired endpoints (`frank-npu`, `frank-small`, and every prior runtime) are in +the [runbook index](#runbook-index). The full swap history lives in +`serverAppConfigs-docs/LLM/index.md` — do not grow it here. ## Repository Layout - `dankserver/` — compose.yaml + Caddyfile + .env.example for the media stack - `dankclaw/` — compose.yaml + Caddyfile + .env.example for the AI stack - `shared/` — homepage, Prometheus, Loki, Alloy, SearXNG configs -- `scripts/` — `deploy.sh`, `deploy-plan.py`, and `deploy-remote.sh` - (service-scoped deployer), LLM serve scripts, systemd units, env tunables, - and bench harnesses -- `old/` — retired PP/NPU-era LLM scripts (reference only) -- `serverAppConfigs-docs/` — knowledge base (Obsidian vault). `serverAppConfigs-docs/LLM/` is the LLM fleet -- `tests/` — bats test suite - -## Documentation Authoring - -The `serverAppConfigs-docs/` vault is the project knowledge base (Obsidian, wikilinks) and is an -OKF (Open Knowledge Format) knowledge bundle. Follow the OKF spec for every doc -you create or edit. - -- **Canonical spec:** `~/.pi/agent/docs/references/okf-spec.md` — the OKF v0.1 - spec in the **system-config knowledge base** at `~/.pi/agent/docs`. The repo's - own `serverAppConfigs-docs/references/okf-spec.md` is deprecated/removed; use the system one. The - spec index and maintenance guide are - `~/.pi/agent/docs/references/index.md` and - `~/.pi/agent/docs/references/okf-maintenance.md`. -- **Read before you touch:** open the relevant `index.md` first - (`serverAppConfigs-docs/index.md` → `serverAppConfigs-docs/LLM/index.md`). Read the whole target file before - editing it. -- **One concept per file**, with YAML frontmatter that has a non-empty `type:` - field (`reference` / `guide` / `configuration` / `research`) plus `description`, - `tags`, and `timestamp`. Split broad topics; keep pages ~1–500 lines. -- **Progressive disclosure:** an `index.md` at each level links its children. - Every `serverAppConfigs-docs/**/*.md` (except `index.md`) must be linked from - `serverAppConfigs-docs/index.md` with `[[Title]]` — the docs-completeness test checks this. -- **Cross-link** siblings as `[[Title]]` wikilinks; cite external claims under a - `# Citations` section. -- **Always update docs** when a task changes behavior, then run - `./tests/run.sh` (the docs-completeness seam is in there) before commit. +- `scripts/` — `deploy.sh` / `deploy-plan.py` / `deploy-remote.sh`, LLM serve + scripts, systemd units, env tunables, bench harnesses +- `old/` — retired LLM scripts (reference only) +- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format) +- `tests/` — bats suite + Playwright e2e ## Git Workflow -All edits flow through git. Do not edit files on the remote machine directly. +All edits flow through git. Never edit files on a remote machine directly. -### Protocol - -1. **Source the config from the target machine** — fetch the live - compose/Caddyfile/serve script from the server where it runs -2. **Place it in this repo** — commit the fetched version (redact secrets - into `.env`) -3. **Edit here** — make all changes locally in the working tree -4. **Run the tests** — `./tests/run.sh` must pass -5. **Commit and push** — push to origin (Gitea on Dankclaw) -6. **Pull on the target** — `git pull` on the remote machine -7. **Deploy** — `scripts/deploy.sh` pulls, plans, and dispatches only the - changed service(s), then checks the selected service health +1. **Source** the live config from the target machine +2. **Commit** the fetched version here (redact secrets into `.env`) +3. **Edit** locally +4. **Test** — `./tests/run.sh` must pass +5. **Commit and push** — origin is Gitea on Dankclaw +6. **Pull** on the target (`deploy.sh` does this) +7. **Deploy** — `scripts/deploy.sh` plans and dispatches only changed services ## Deployment -Use `scripts/deploy.sh` to deploy committed changes to a target. The planner -maps changed paths to individual services; the remote runner restarts only the -selected service and records its successful revision outside the checkout. - ``` ./scripts/deploy.sh dankclaw ./scripts/deploy.sh dankserver @@ -136,599 +63,350 @@ selected service and records its successful revision outside the checkout. ./scripts/deploy.sh dankclaw --dry-run ``` -A documentation-only change is a no-op. The LLM services are selected only by -their own unit/env/serve files or the shared chat template, so unrelated -container changes cannot interrupt inference. `frank-large` is selected by its -Quadlet file under `dankclaw/frank-large/quadlet/`. - -Set `FRANK_SUDO_PW` for headless runs on a host without passwordless sudo -(remote sudo reads it from stdin via `sudo -S`). The value is never stored -in this repo. +The planner maps changed paths to services; the remote runner restarts only +the selected service and records its revision outside the checkout. +Documentation-only changes are a no-op. LLM services are selected only by +their own unit/env/serve files or the shared chat template — unrelated +container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless +runs on a host without passwordless sudo; the value never enters the repo. ### Deployment completeness (MANDATORY) -A deployment is **not complete** when `deploy.sh` exits green. It is complete -only when BOTH of the following pass: +A deploy is **not complete** when `deploy.sh` exits green. It requires: -1. **Unit/config tests** — `./tests/run.sh` (all seams green), and -2. **Playwright end-to-end tests** — `tests/e2e/run.sh` drives the LIVE fleet - (homepage, Anubis/Authentik guard chain, guarded vhosts) in a real - browser. As of 2026-09-12 this runs **automatically** as the `[3/3]` gate - at the end of `deploy.sh`: a behavior-affecting deploy that fails the e2e - suite exits `DEPLOY INCOMPLETE` and must not be treated as good. The suite - stays out of `./tests/run.sh` (it needs live hosts plus the `qa-e2e` - credentials in `tests/e2e/.env.local`). Bypass only with - `deploy.sh --skip-e2e` in an emergency. +1. `./tests/run.sh` — all seams green, and +2. the Playwright e2e suite (`tests/e2e/run.sh`) — runs **automatically** as + the `[3/3]` gate at the end of `deploy.sh` against the LIVE fleet; a + behavior-affecting deploy that fails it exits `DEPLOY INCOMPLETE`. Bypass + only with `--skip-e2e` in an emergency. ```mermaid graph LR - A[Edit locally] --> B[Tests pass] - B --> C[Commit] - C --> D[Push to origin] - D --> E[deploy.sh computes service plan] - E --> F[Target pulls with ff-only] - F --> G[deploy-remote.sh dispatches selected service] - G --> H[Per-service health check] - H --> I[Playwright e2e gate on live fleet] - I --> J{e2e green?} - J -->|yes| K[Deploy complete] - J -->|no| L[DEPLOY INCOMPLETE] + A[Edit locally] --> B[Tests pass] --> C[Commit + push] + C --> D[deploy.sh plan] --> E[Target pulls ff-only] + E --> F[Dispatch selected service] --> G[Per-service health] + G --> H[Playwright e2e gate] --> I{green?} + I -->|yes| J[Deploy complete] + I -->|no| K[DEPLOY INCOMPLETE] ``` ## Boot Persistence & Self-Healing (MANDATORY) -Every Podman container governed by this project that is successfully deployed -and proven working MUST have a self-healing, boot-persistent Quadlet unit. A -container that runs only from `compose.yaml` (or a manual `podman run`) dies -silently at reboot: rootless Podman does not auto-start bare containers, and -`restart: unless-stopped` does not survive the explicit stop that shutdown -performs. Do not report such a service as "up" without its unit. +Every deployed, working Podman container MUST have a self-healing, +boot-persistent Quadlet unit. A container that runs only from `compose.yaml` +or a manual `podman run` dies silently at reboot — rootless Podman does not +auto-start bare containers, and `restart: unless-stopped` does not survive the +explicit stop that shutdown performs. Never report such a service as "up". -Each container's unit must provide: +Unit requirements: - **Boot persistence** — `[Install] WantedBy=default.target`, installed to - `~/.config/containers/systemd/.container` (generating - `.service`) and enabled in the user manager - (`systemctl --user enable .service`). Linger must be on for the deploy - user (`loginctl show-user dank -p Linger` → `Linger=yes`) so user units start - at boot with no login session. -- **Self-healing** — `[Service] Restart=always` (or `Restart=on-failure`) with - a short `RestartSec`, so a crash or OOM brings the container back without - operator action. Add a quadlet health check where the service has one. + `~/.config/containers/systemd/.container`, enabled in the user + manager, with linger on for the deploy user (`loginctl show-user dank -p + Linger` → `Linger=yes`). +- **Self-healing** — `Restart=always` (or `on-failure`) with a short + `RestartSec`; quadlet health check where the service has one. - **Repo provenance** — the `.container` file lives in this repo - (`dankclaw//quadlet/` or `dankserver//quadlet/`), is installed by - the service-scoped deployer, and is covered by the quadlet test seam. The - host-side unit is generated state; the repo file is the source of truth. + (`//quadlet/`), is installed by the deployer, and is covered by + the quadlet test seam. The host-side unit is generated state; the repo file + is the source of truth. -Verification before a service is declared deployed: -`systemctl --user is-enabled .service` → `enabled`, -`systemctl --user is-active .service` → `active`, and the container -answers its health check. A running container with no unit is a gap: write the -unit, test it, deploy it. +Before declaring a service deployed: `systemctl --user is-enabled ` → +`enabled`, `is-active` → `active`, and the container answers its health check. +A running container with no unit is a gap: write the unit, test it, deploy it. -Incident that established this rule (2026-08-26, Dankclaw reboot): Caddy ran -for weeks as a bare compose container and never came back after reboot; the -Pi-hole unit existed but crash-looped on a sysctl conflict (a later-sorted -drop-in reset `net.ipv4.ip_unprivileged_port_start` to 80), taking LAN DNS -down with it. - -**Observability gap (found 2026-09-11, fixed the same day):** rebooting -Dankclaw for the `amd_iommu=off` change exposed five services that ran only -from `dankclaw/compose.yaml` with no Quadlet unit — `prometheus`, `grafana`, -`loki`, `alloy` and `node-exporter`. They did not come back and had to be -restored by hand. They are now rootless Quadlet units under -`dankclaw/observability/quadlet/` (`./scripts/deploy.sh dankclaw --service -`), and their compose definitions were removed so only one owner exists -per container name. The three HTTP UIs publish on loopback only; off-host -access is the guarded vhost. -Note that the stack is RUNNING BUT UNCONFIGURED: Grafana has no datasources -and no dashboards, and Prometheus has no rules and no Alertmanager. See +This rule exists because reboots found gaps twice (2026-08-26: Caddy as a +bare compose container never returned; 2026-09-11: five observability +services had no units). The observability services now have units under +`dankclaw/observability/quadlet/` and their compose definitions were removed — +one owner per container name. The stack is RUNNING BUT UNCONFIGURED (no +Grafana datasources/dashboards, no Prometheus rules); see `serverAppConfigs-docs/Infrastructure/Observability.md`. ## Testing -Tests are mandatory before commit. `bats` must be installed on the -development machine. +Tests are mandatory before commit (`bats` required on the dev machine): ``` ./tests/run.sh ``` -Test seams: compose validity, env consistency (no leaked secrets), Caddyfile -validity, quadlet units, docs completeness, script syntax (all of `scripts/`, -including the LLM serve scripts), LLM unit integrity (units point at files in -this repo), LLM secret hygiene, deployment flow, and service-scoped deployment -planning/dispatch. See [serverAppConfigs-docs/Architecture/Testing Policy.md](serverAppConfigs-docs/Architecture/Testing%20Policy.md). +Seams: compose validity, env consistency (no leaked secrets), Caddyfile +validity, quadlet units, docs completeness, script syntax, LLM unit +integrity, LLM secret hygiene, deployment flow, service-scoped planning. +See `serverAppConfigs-docs/Architecture/Testing Policy.md`. ## Secret Rules -- Never commit real secrets. The live Gitea database password and the live - sudo password are test-enforced as leak checks. -- `.env` files are git-ignored. Commit only `.env.example` with placeholders. - The LLM env tunables in `scripts/` are tracked by exception (`!scripts/*.env`). - They hold paths and numbers, never credentials. -- Do not document passwords in this repo, including sudo passwords. SSH keys - are the auth mechanism. Headless sudo passes the password via the - `FRANK_SUDO_PW` environment variable. The value never enters the repo. - +- Never commit real secrets. The live Gitea DB password and sudo password are + test-enforced leak checks. +- `.env` files are git-ignored; commit only `.env.example` with placeholders. + Exception: `scripts/*.env` LLM tunables (paths and numbers, never + credentials). +- Do not document passwords anywhere in the repo. SSH keys are the auth + mechanism; `FRANK_SUDO_PW` covers headless sudo and never enters the repo. ## External Exposure Policy (MANDATORY) -Every service reverse-proxied through Caddy is either behind the auth chain -(Anubis :8923 → Authentik embedded outpost) or explicitly designated -internal-only — no public DNS name, no vhost. New services default to -internal-only until guarded. +Every Caddy-proxied service is either behind the auth chain (Anubis :8923 → +Authentik embedded outpost) or explicitly internal-only — no public DNS name, +no vhost. New services default to internal-only until guarded. -When guarding a new vhost, follow this order — provider first, then route: +When guarding a new vhost — provider first, then route: -1. Create the outpost proxy provider FIRST (`external_host` = the exact vhost - URL including trailing slash; `internal_host = http://caddy:8017`). If the - vhost flips before the provider exists, authenticated traffic hits the - outpost error page instead of the app. +1. Create the outpost proxy provider FIRST (`external_host` = exact vhost URL + with trailing slash; `internal_host = http://caddy:8017`). If the vhost + flips first, authenticated traffic hits the outpost error page. 2. Flip the vhost to `reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`, and add the post-auth - `@ host ` handler on the `http://:8017` listener routing to the - real upstream. -3. The Anubis guard set, the Caddy TLS edge, and the outpost provider list - must stay in lockstep — they change together in one deploy. -4. Service-to-service traffic (e.g., Alloy → Loki push) targets internal - addresses (`loki:3100`), never the public hostname — non-browser clients - cannot solve challenges. + `@ host ` handler on the `http://:8017` listener routing to + the real upstream. +3. Anubis guard set, Caddy TLS edge, and outpost provider list change + together in one deploy. +4. Service-to-service traffic targets internal addresses (`loki:3100`), never + the public hostname — non-browser clients cannot solve challenges. + +Exempt categories (document the reason in the vhost comment): the IdP itself +(`auth.`), git smart-HTTP endpoints, services with their own complete auth. -Exempt-by-construction categories stay exempt with a documented reason in the -vhost comment: the IdP itself (`auth.` — its login flow must never sit behind a -challenge), git smart-HTTP endpoints (git clients cannot solve JS challenges), -and services with their own complete auth. ## Deletion & Trash Policy (MANDATORY) -Never run `rm` to delete files. The command deletes permanently. Move every -item you delete into trash instead. The trash keeps the deletion recoverable -and reviewable. +Never `rm`. Move deletions into trash so they stay recoverable: -1. In-repo files: move the item to `.AGENTTRASH/` beside its directory. -2. Home and other paths: create a dated staging directory - `~/.AGENTTRASH/YYYYMMDD-topic` and move each item there. -3. List the staged items and their sizes for the user. -4. Wait for explicit user permission before you empty the trash. +1. In-repo files → `.AGENTTRASH/` beside the item's directory. +2. Elsewhere → dated staging dir `~/.AGENTTRASH/YYYYMMDD-topic`. +3. List staged items and sizes; wait for explicit user approval to purge. -Rules: - -- `rm`, `rm -rf`, and `git clean -f` are forbidden without explicit user - permission. -- `git * --no-verify` is forbidden. -- A moved item stays in the trash until the user approves its purge. -- The trash is a temporary hold, not a long-term archive. +`rm`, `rm -rf`, and `git clean -f` are forbidden without explicit permission. +`git * --no-verify` is forbidden. The trash is a temporary hold, not an +archive. ## Benchmarking Policy (MANDATORY) -**STOP the production service BEFORE running any benchmark on a target.** -These servers serve real users (about 3 people). Running a bench while the -service is up starves the GPU and corrupts results. Restart the service when -the bench finishes. +**STOP the production service BEFORE benchmarking** — these servers serve +real users, and a bench beside a live service starves the GPU and corrupts +results. Use the runtime's own harness: `llama-bench` (llama.cpp), `vllm +bench` (vLLM), halogen's `bench`/`sweep` modes for frank-large. Rolling your +own bench requires asking first. Scratch ports are not used on LLM hosts — a +test process cannot share the single GPU anyway; validate against the service +itself. -ALWAYS bench using llama-bench for llama.cpp or vllm bench for vllm runtimes. -DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE OFFICIAL BENCHES. ASK THE USER BEFORE DEVIATING FROM THIS DIRECTIVE. +1. Stop and mask (self-healing would restart it): + - Dankclaw `frank-large` (rootless — no sudo): + `systemctl --user stop frank-large && systemctl --user mask frank-large` + - Dankserver `frank-medium` (system unit): + `sudo systemctl stop frank-medium && sudo systemctl mask frank-medium` +2. Bench. +3. Unmask and start; verify `curl http://192.168.0.:7770/health`. -1. Stop the service before any benchmark: - `systemctl --user stop frank-large` (Dankclaw — a **rootless** Quadlet - service, so drop the `sudo`) or `sudo systemctl stop frank-medium` - (Dankserver — still a system unit). - Then mask it to prevent the self-healing unit from restarting: - `systemctl --user mask frank-large` (Dankclaw) or - `sudo systemctl mask frank-medium` (Dankserver). - **CRITICAL WARNING:** Do NOT use `systemctl disable` on the **system** units. - On these hosts, `systemctl disable` doesn't just remove symlinks — it **deletes - the unit file itself** from `/etc/systemd/system/`. You'd have to manually - copy the unit file back from this repo, run `daemon-reload`, then enable and - start. Mask (`systemctl mask`) creates a symlink to `/dev/null` instead, which - is safe and reversible. -2. Run the bench with the runtime's own harness: `llama-bench` for llama.cpp, - `vllm bench` for vLLM, and for frank-large's halogen-flash-server the - image's own `bench`/`sweep` modes (`bench [drafters] [max_tokens] [effort] - [reps]`) — `llama-bench` and `vllm bench` do not apply to it. Scratch ports are not used for tests or benches on the LLM - hosts — a test process cannot share the single GPU with the running service - anyway (the model + KV fill the card), so changes are validated directly - against the service itself. -3. Restart the service when the bench completes: unmask, then start: - `systemctl --user unmask frank-large && systemctl --user start frank-large` - (Dankclaw) or `sudo systemctl unmask frank-medium && sudo systemctl start - frank-medium` (Dankserver). -4. Verify health before reporting: - `curl http://192.168.0.235:7770/health` or - `curl http://192.168.0.143:7770/health`. +**CRITICAL:** never `systemctl disable` a system unit on these hosts — it +deletes the unit file from `/etc/systemd/system/`, not just the symlink. +`mask` symlinks to `/dev/null` and is safe/reversible. ## Model Config (harness) -The harness reaches the fleet through `/home/dank/.omp/agent/models.yml` — -verified live 2026-09-11; the older `/home/dank/.pi/agent/models.json` path -this section used to name no longer exists. Both files live outside the -repository. Each provider is one host endpoint and each model `id` is the -served alias: +The harness reaches the fleet through `/home/dank/.omp/agent/models.yml` +(outside the repo). Each provider is one host endpoint; each model `id` is +the served alias: `frank-medium` → `192.168.0.235:7770`, `frank-large` → +`192.168.0.143:7770`. Per-role routing lives in +`/home/dank/.omp/agent/config.yml` under `modelRoles`. -| alias | endpoint | -|---|---| -| `frank-medium` | Dankserver `192.168.0.235:7770` | -| `frank-large` | Dankclaw `192.168.0.143:7770` | -| `gemma4-it:e4b` | *(removed 2026-09-11 with `frank-npu`)* | +Resolution rules: the default model resolves as a `defaultProvider` + +`defaultModel` pair (the provider must serve the model); a subagent override +uses a bare model `id`, which resolves only when unique across providers. -Two resolution rules apply (verified in `@earendil-works/pi-coding-agent` -`model-resolver.js`): +When renaming a service, retiring an endpoint, or moving a model: update the +provider block, model `id`, `contextWindow`, `maxTokens`, `input` in +`models.yml`, then fix every `modelRoles` entry that named the old endpoint — +a role pointing at a removed provider fails at call time. `frank-large` +declares `input: [text, image]` (halogen vision tower). There is **no audio +input anywhere in the fleet** (left with `frank-npu`). -- The default model resolves as a `defaultProvider` + `defaultModel` pair. - The `defaultProvider` must name the provider that serves the `defaultModel`. -- A subagent override uses a bare model `id`. A bare `id` resolves only when - it is unique across all providers. +## Active Services -Per-role routing lives in `/home/dank/.omp/agent/config.yml` under -`modelRoles` (`smol`, `tiny`, `task`, `plan`, `designer`, `advisor`, `slow`, -`vision`, …). When you rename a service, retire an endpoint, or move a model -to a new host, update the provider block, the model `id`, `contextWindow`, -`maxTokens` and `input` in `models.yml`, then fix every `modelRoles` entry -that named the old endpoint — a role pointing at a removed provider fails at -call time. `frank-large` declares `input: [text, image]` because halogen runs -its vision tower. +### frank-large — halogen-flash-server (Dankclaw) -## Runtime & Troubleshooting +`ghcr.io/peonist-ai/halogen-flash-server:0.5.8` as a rootless Quadlet +(`dankclaw/frank-large/quadlet/frank-large.container` → `frank-large.service` +in the user manager), serving `peonist-ai/halogen-qwen3.8-flash-next`. +One container, two halves: engine on container loopback :8730 (unauthenticated, +NEVER published) and OpenAI front-end on :8731 published as host **7770**. -### frank-npu — FLM NPU endpoint (Dankclaw, RETIRED 2026-09-11) +Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=401408`, +`HALOGEN_KV_SLOTS=3`, `HALOGEN_MAX_TOK=32768`, `HALOGEN_HOST_RESERVE_GIB=15`, +`HALOGEN_VISION_TOWER=1`. Everything else upstream default. Sizing history +and the flag-by-flag rationale live in the runbook — do not re-derive here. -Retired by operator decision together with the kernel cmdline change that -bought 13–16 % of prefill for `frank-large`: with `amd_iommu=off` the XDNA2 -driver cannot bind SVA/PASID, so every open of `/dev/accel/accel0` fails with -`err=-19`. The unit is stopped and masked, the service files are staged in -`.AGENTTRASH/` (`*.flm-npu`), and the providers that named it were removed -from both the harness and Hermes. **Audio input left the fleet with it and -has no replacement**; image input survives on `frank-large` through halogen's -vision tower. Route map: Hermes `vision` → `frank-large`, Hermes -`title_generation` → `frank-medium`, harness `vision` → `frank-large`, harness -`tiny` → `frank-medium`, harness `task` → `frank-large` at medium effort. +Invariants that survive any config change: -The retired runbook keeps the full record, including the original IOMMU/SVA -root cause: [serverAppConfigs-docs/LLM/frank-npu.md](serverAppConfigs-docs/LLM/frank-npu.md). -The current kernel posture and its revert path: -[serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md](serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md). +- **Pool and reserve are a matched pair.** Budget = `121.2 host − 67.7 + weights − reserve`; that one budget carries KV pool + prefill arena + the + 1.97 GiB vision tower. The reserve is **not an allocation** — raising it + cannot free host RAM, it can only shrink the pool. To free host RAM, cut + `HALOGEN_KV_POOL_POSITIONS` or `HALOGEN_MAX_TOK`. +- **`free` lies on this host.** The ~68 GiB pinned weights count as + reclaimable page cache, so `free`/`MemAvailable`/Grafana overstate free + memory by the model size. Believe the server's startup ledger. A co-tenant + running the box dry looks like minutes of full-CPU stalls with no disk + activity — not a crash. +- **Client-visible contract:** no server-side sampler defaults (no + `temperature` ⇒ greedy); per-request budget defaults to 8192 and **includes + thinking** (running out mid-thought ⇒ `finish_reason: "length"`, empty + content); cap 65536; a request reserving prompt + `max_tokens` beyond the + pool queues, then fails `503 engine_busy` after the 3600 s timeout. -### halogen-flash-server (Dankclaw, ACTIVE since 2026-09-11) +Runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`. -The large service runs `ghcr.io/peonist-ai/halogen-flash-server:0.5.8` as a -**rootless Quadlet container** -(`dankclaw/frank-large/quadlet/frank-large.container` → generated -`frank-large.service` in the user manager), serving -`peonist-ai/halogen-qwen3.8-flash-next`: Qwen3.8-Flash-Next as a 115.55 GiB -w4b `.hgn` checkpoint plus its quality overlay sidecar. One container runs -both halves — the engine on container loopback :8730 (unauthenticated, never -published) and the OpenAI front-end on :8731 published as host **7770** — -with `HALOGEN_MODEL_ID=frank-large`, so no client wiring changed. +### frank-medium — ROCmFPX llama.cpp (Dankserver) -Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=401408` -(**trimmed 2026-09-13** from the 524,288 the 09-12 doubling deployed — one -full 262k conversation resident beside a ~139k one, or three at 131k; hands -~3.4 GiB back for the ~28 co-tenants after the doubled pool left only -~7.1 GiB and the engine warned of co-tenant stalls), `HALOGEN_KV_SLOTS=3` -(lowered from 4 on 2026-09-12 to the real fleet size; 111 MiB of O(1) state -each), `HALOGEN_MAX_TOK=32768`, `HALOGEN_HOST_RESERVE_GIB=15`, -`HALOGEN_VISION_TOWER=1`, and -`HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next` for the first-start -fetch; everything else is upstream's default. **Pool and reserve are a matched -pair:** the budget is `121.2 host − 67.7 weights − reserve`, and that one -budget carries the KV pool, the prefill arena and the 1.97 GiB vision tower -together. The reserve is **not an allocation** — 15 and 16 were both measured -holding 103.5 GiB with 7.1 GiB left on the host, so a bigger reserve cannot -free memory for co-tenants, it can only remove the pool. Against the old -524,288 pool the reserve ceiling was 16 (engine needs budget ≥ 36.5; reserve -18 yields 35.5 and silently clamps); at 401,408 the need is ~33.4 so the -ceiling loosens, but a bigger reserve still buys nothing. To free host RAM, -cut `HALOGEN_KV_POOL_POSITIONS` (262144 hands back another ~3.6 GiB) or -`HALOGEN_MAX_TOK` (16384 hands back 8.4). Measured at 401,408 (2026-09-13 -deploy ledger): 100.0 GiB held (68.0 weights + 11.0 KV + 21.0 working), ~9.1 -GiB / 1396 contiguous 2 MiB blocks left on the host, engine need printed -~31.5 GiB against the 38.5 budget, and the co-tenant stall NOTE no longer -prints — confirm on the boot ledger, not here. If a -co-tenant still runs it dry, frank-large and its neighbours stall for -minutes at full CPU with no disk activity — not a crash, and invisible to -`free`, which still reads ~74 GiB available. The weights stay pinned (~68 GiB) -and the kernel counts those locked pages as reclaimable page cache, so `free`, -`MemAvailable` and anything built on them (Grafana) overstate free memory by -the size of the model — believe the server's own startup ledger, not the -dashboards. +charlie12345/ROCmFPX llama.cpp `c49ebdb`, built for gfx1201 with the fork's +self-contained ROCm 7.14 toolchain (RPATH-bundled — host ROCm packages never +participate). Serves `kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF` +(ftype 115; stock llama.cpp cannot load it) with `--split-mode layer` across +both R9700s, 131072 ctx / 1 slot, f16 KV, flash-attn on, MTP draft head +pinned to ROCm1, reasoning on. Env `scripts/frank-medium.env`, serve +`scripts/serve-frank-medium.sh`. -Client-visible contract: **no server-side sampler defaults** (a request that -sends no `temperature` decodes greedy); the per-request budget defaults to -8192 and **includes thinking**, so a budget that runs out mid-thought returns -`finish_reason: "length"` with empty content; the cap is 65536; and a request -reserves prompt + `max_tokens` positions against the KV pool, so a request -larger than the pool is never admitted — it queues and then fails with -`503 engine_busy` after the 3600 s queue timeout. +Runbook: `serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md`. -Runbook: [serverAppConfigs-docs/LLM/halogen-flash-server.md](serverAppConfigs-docs/LLM/halogen-flash-server.md). +## Key Lessons Learned -### kingjones Qwen3.8-Flash-Next STRIX on kingjones30/ROCmFPX (Dankclaw, PRIOR — rollback target) +Durable operational lessons from the fleet's history. Full detail in the +linked runbooks. -The large service served -`kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` (public, 113.47 -GiB single file, `Q4_0_ROCMFP4_STRIX` = file_type 105) from -`/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/` with -the card's **pinned runtime**: the `kingjones30/ROCmFPX` fork -(charlie12345/ROCmFPX + qwen4exp-on-rocmfpx-d3ca537 patch — ROCmFP4 -types and qwen4exp arch in one tree), built at -`/home/dank/GitPackages/kingjones-ROCmFPX/build` with -`-DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON --DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_WEBUI=OFF` (WebUI -provisioning needs npm/HF-bucket and aborts the link at 98% without -the OFF flag). Args: our validated profile — 401408 context / 2 slots, -b 4096 / ub 512 (ub 512 matches the card's re-measurement), t 16 / tb -16, f16 KV, `--fit off`, `--flash-attn on` (works UNPATCHED on this -model — the QSA F16 ggml_fill assert that needed -`scripts/patches/qwen4exp-qsa-f32-mask.patch` is specific to the -agentionai FAST layer set; if a future build aborts with -`GGML_ASSERT(a-type == GGML_TYPE_F32)`, port that fix), reasoning on -(Sharp template, deepseek format), no speculative decoding (card ships -no spec flags; Flash-Next MTP heads do not load on this loader family). -Two hard card constraints: **mmap must stay on** (`--no-mmap` forces -the 50.66 GiB Q8_0 PLE table into anonymous memory → silent cgroup -kill, only dmesg evidence; llama.cpp's "use_mmap=1" log line is -correct here — mmap pages ARE page cache) and **never pin the PLE -tensor with `-ot`** (23 → 13.4 t/s). The 51.2B-param PLE table never -enters GPU memory (~20 bytes/token-file per 500 decoded). On-host file -sha256 verified == HF LFS oid `c6770d74...49fa`. Its env and serve script -(`scripts/frank-large.env`, `scripts/serve-frank-large.sh`) are staged in -`.AGENTTRASH/`, suffixed `.llama.cpp`. -Runbook: -[serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md](serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md). +**ROCm / GPU** -**Serving status (2026-08-30):** swapped from the agentionai FAST -deployment. Validated live before the systemd handoff: clean load, -health 200, reasoning completions, and an operator session's ~111k -token prefill at 213–232 t/s (card rung: 196 at 131k). Note llama.cpp -clamps n_ctx_train 262144 → per-slot 131072 with parallel 2. +- **Layer split, never tensor, on the RCCL-less ROCmFPX builds.** Tensor + parallel falls back to a butterfly allreduce and collapses prefill ~7x + (measured 271.8 vs 1991 t/s). Pin the MTP draft head to ROCm1 — an + unpinned draft under tensor mode aborts the KV build. +- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the + ~50 GiB PLE table into anonymous memory → silent cgroup kill, dmesg is the + only evidence. mmap pages ARE page cache. Never pin the PLE tensor with + `-ot` (23 → 13.4 t/s). +- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve + scripts export env files with `set -a`, so a tunable named `TEMP` leaks + into HIP (`TEMP=1.0` sent the JIT cache to path `1.0` and segfaulted every + load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a tunable + these; check with + `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"`. +- **`-DGGML_CUDA_FA_ALL_QUANTS=ON` is required** for q8_0 KV + flash + attention on CUDA builds; without it prefill collapses ~12x (95 vs 1300+ + t/s, measured 2026-08-17). +- **Flash-Next MTP heads do not load on any current loader** — they need full + per-layer `blk..hc_attn_norm.weight` and ship only `blk.48` (verified + 3x). Don't re-attempt without new loader support. +- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on + qwen4exp needs `scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on + the agentionai FAST layer set; the kingjones STRIX set works unpatched. +- **GPU budget on Dankclaw is the rocm-smi figure** (64 GiB, SVM-backed). + `free`/`dmidecode` show the CPU-side carve-out — intentional, do not + change. -### agentionai ROCmFP4-FAST imatrix llama.cpp (Dankclaw, prior, 2026-08-29 to 2026-08-30) +**Load / crash diagnosis** -Historical — rollback target. `agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` -(87.06 GiB, Q4_0_ROCMFP4_FAST imatrix, v2/ joined-PLE layout only — the -root `-ple16` per-head split aborts this loader family) on the charlie -`build-qwen38` tree (`/home/dank/GitPackages/ROCmFPX`, PR #27742 -merged) at 32768/1, ub 4096. That tree REQUIRES -`scripts/patches/qwen4exp-qsa-f32-mask.patch` for `--flash-attn on` -(F16-kq_mask ggml_fill abort otherwise). Flash-Next MTP heads do not -load on any current loader (require full per-layer -`blk..hc_attn_norm.weight`; heads carry only `blk.48` — verified -3x). Binary and payload retained. Runbook: -[serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md](serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md). +- Model load is slow (minutes cold); health returns 503 while loading; do + not kill the process. `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). On + halogen, the FIRST request after a (re)start can go silent ~45–60 s + (health `engine_unresponsive`, 503s) and then self-recovers — do not + restart over it. +- Crash-loop procedure: (1) `journalctl -u -n 100` + kernel line via + `journalctl -k` (segfault at address 8 in `libamdhip64` = NULL unref in + HIP); (2) run serve script, then raw binary — different outcome isolates + the env file; (3) `gdb -batch -ex run -ex "bt 25" --args bash + scripts/serve-.sh` — llama-server buffers stdout and loses the log + tail on SIGSEGV; trust gdb, not the log; (4) `amdgpu: SVM mapping failed` + spam is a symptom, not a root cause. +- Journal signatures: client abort logs `W srv stop: cancel task, id_task = + N`; natural stop logs only `slot release` with `truncated = 0`. The server + never preempts a running generation; excess tasks queue when slots are + busy. -### kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT on ROCmFPX llama.cpp (Dankserver, ACTIVE since 2026-09-15) +**Host memory / swap** -The medium service serves `kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF` -(public, 26.28 GiB, 8.39 bpw, `Q8_0_ROCMFPX_AGENT` ftype 115 — stock -llama.cpp cannot load it) from -`/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` with -charlie12345/ROCmFPX llama.cpp `c49ebdb`, built for **gfx1201** with the -fork's self-contained ROCm 7.14 toolchain -(`env JOBS=8 ./scripts/build-rocmfp4-rocm714-local.sh` in -`/home/dank/GitPackages/charlie-ROCmFPX`; the binary bundles its ROCm runtime -via RPATH, so the host's ROCm packages never participate). `--split-mode -layer` across both R9700s — tensor parallel loses 7x prefill on this -RCCL-less build — 131072 context / 1 slot, f16 KV, flash-attn on, MTP draft -head (n_max 6, p_min 0.6, `--spec-draft-device ROCm1`), reasoning on -(deepseek format), sampler defaults from the checkpoint's instruct table. -Measured 2026-09-15: llama-bench pp4096 1991 t/s / tg128 18.8 t/s (tensor: -271.8 / 26.26; single card: 1160.9 / 19.42); served with MTP 35.7–46.2 tok/s -and 60–69 % draft acceptance; prefix caching verified working with the draft -head. Env `scripts/frank-medium.env`, serve `scripts/serve-frank-medium.sh`. -Runbook: -[serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md](serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md). +- Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff` + of the old one, or live pages must land in RAM that `MemAvailable` + overstates. Dankclaw swap (64 GiB at `/swap.img`, fstab-driven) protects + the co-tenants, not frank-large — pinned weights don't swap, and paging + the KV pool is worse than the OOM it prevents. +- Host swap state is invisible to `deploy.sh` (no repo file) — check + `swapon --show` before reasoning about headroom. -### Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on FreeToken (Dankserver, RETIRED 2026-09-15) +**Runtime-specific guards** -Retired with the RTX 3090: FreeToken/CUDA cannot run on the replacement -R9700 cards, and the FreeToken-era unit was archived to -`~/.AGENTTRASH/20260915-frank-small-retire/host-units/`. Historical record: +- **Kairic Edge** (if ever re-deployed): the pinned CK commit needs + `third_party/composable_kernel/include/ck/config.h` generated manually + (untracked; `git clean` removes it), and its greedy-argmax fastpath rejects + non-greedy requests and reasoning budgets. Re-test chat before production. +- **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch; + stock binaries fail the draft load with a tensor-shape mismatch. -The small service serves `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` -(public, ~35 GB, 26 safetensors shards) from -`/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8/` with -FreeToken v0.1.2 (`/home/dank/freetoken/.venv`, `ft serve`). Native -modelopt block-fp8 (e4m3, dynamic, 128×128 weight_block_size, -un-fused per-expert layout) — same format and loader path as the -official Qwen3.6-35B-A3B-FP8 it replaced on 2026-08-30, so the entire -memory/pool plan carries: experts host-RAM with 20% LRU GPU residency, -paged KV (only the 10 full-attention layers page; 30 GDN layers use -constant-size mamba slots), ~3.4 GiB VRAM headroom at boot, and the -validated `--memory-num-gpu-blocks-experts 2048 ---memory-num-gpu-tokens-kv-cache 453005 --max-num-seqs 4` pool triple -(never grow the expert pool). 262144 context, reasoning on (the -checkpoint ships the froggeric fixed chat template — no override). -FreeToken's API REQUIRES an explicit `"model"` field in chat requests -(422 without it). Env `scripts/frank-small.env`, serve -`scripts/serve-frank-small.sh`. Runbook: -[serverAppConfigs-docs/LLM/freetoken-frank-small.md](serverAppConfigs-docs/LLM/freetoken-frank-small.md). +**Ops** -**Serving status (2026-08-30):** swapped the model within the FreeToken -runtime (official FP8 → Huihui abliterated FP8). ~40 tok/s verified on -a 282-token completion post-deploy. Prior: official Qwen FP8 -(2026-09-02, runtime swap), Tiel-Coder-35B on llama.cpp (2026-09-01), -ik_llama.cpp Ornith CRACK (retired 2026-09-02). +- Never `pkill -f` a pattern that appears in your own command line — it kills + your ssh session. Use `pkill -x `. +- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in + committed scripts (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts + without it. +- ComfyUI on Dankclaw runs CPU-only (`--cpu`): no ROCm wheels for gfx1151, + and CUDA images are wrong for the hardware. +- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its + ports off the gluetun service. +- `amd_iommu=off` on Dankclaw (bought 13–16 % prefill for frank-large, + 2026-09-11) permanently retired the XDNA2 NPU: the driver's SVA/PASID bind + needs an enabled IOMMU (`/dev/accel/accel0` fails `err=-19`). Revert path + in `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md`. +- Rootless Quadlet containers run with `--replace --rm`: `podman logs` covers + only the CURRENT instance; history across recreations lives in + `journalctl --user -u .service`. Read rootless storage WITHOUT sudo + (per-user storage — `sudo podman logs` finds nothing). -### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, prior v3, 2026-08-18 to 2026-08-25) +## Runbook Index -The v4 runtime serves `cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (`eemin` namespace, -public) from `/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` -(charlie12345/ROCmFPX commit `c49ebdbd` built with `-DGGML_HIP_ROCMI4_W4A4=ON` -against system ROCm 7.2.4 / amdclang 22). The model is `Q4_0_ROCMI4` with an -embedded NextN/MTP head (no separate draft file); the gfx1151 native IU4 WMMA -instruction batch-verifies proposals. Profile: strict Qwen MTP -(`--spec-mtp-strict-qwen`, n_max 16, p_min 0.60, backend sampling, all layers -on ROCm0), `-t 16 -tb 32`, batch 512 / ubatch 256 (ubatch swappable via the -ubatch bench on 8k context), f16 KV, full 262144 context / 1 slot, TEMPERATURE -0.7 / TOP_P 0.95, reasoning on (deepseek, budget -1), `--no-mmap`. Two-file -payload (~14.4 GiB) in `/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP/`. Full -runbook: -[serverAppConfigs-docs/LLM/eemin-rocmi4-mtp.md](serverAppConfigs-docs/LLM/eemin-rocmi4-mtp.md). +Active: +| Service | Runbook | +|---------|---------| +| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` | +| frank-medium (ROCmFPX, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md` | +| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | +| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | -### Kairic Edge llama.cpp (Dankclaw, RETIRED experiment 2026-08-22) +Retired (rollback targets and history): -The Kairic Edge llama-server serves `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` -(pinned tag `kairic-edge-qwen38-27b-v1`). It was deployed and reverted to v3 -the same day (operator decision). The runtime routes qualified prompt and -verification shapes through the RDNA 3.5 native IU4 matrix instruction. +| Runtime | Host | Window | Runbook | +|---------|------|--------|---------| +| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` | +| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` | +| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` | +| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` | +| CIRU ActiveFPX v3 / PromptForge | Dankclaw | through 2026-08-22 | `ciru-activefpx-promptforge.md` | +| DFlash2 Vulkan | Dankclaw | 2026-08-21 (same-day revert) | `dflash2-large-vulkan.md` | +| FastFlowLM NPU (`frank-npu`) | Dankclaw | twice, retired 2026-09-11 | `frank-npu.md`, `FastFlowLM NPU Inference.md` | +| FreeToken (official + Huihui FP8, `frank-small`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` | +| FastMTP patched llama.cpp | Dankserver | retired 2026-08-18 | `qwen3.8-spec-matrix.md` | +| ik_llama.cpp Ornith CRACK | Dankserver | retired 2026-09-02 | `ikllama-ornith-dankserver.md` | -**Release gap — CK `config.h`:** the pinned CK commit ships only -`config.h.in`. The release script and guide never generate -`third_party/composable_kernel/include/ck/config.h`. Generate it via a CK -CMake configure and copy it in before building (full runbook in -[serverAppConfigs-docs/LLM/kairic-edge.md](serverAppConfigs-docs/LLM/kairic-edge.md)). The file is untracked -in CK git. A `git clean` removes it. Local rebuilds are byte-identical -(`d7625a0c...`). - -**Target greedy argmax fast path:** the pinned runner sets -`LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1`. The fast path rejects non-greedy -requests (400) and the card's own chat sampling is non-greedy, so the -production attempt ran it at 0. It still rejects reasoning budgets unless -compatibility mode is used. Do not re-deploy Kairic without re-testing chat -first. - -**Rollback target:** the v3 model files remain in -`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/` and the v3 binary in -`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`. -Restored to v3 on 2026-08-22 (commit `c60156f`). - -### DFlash2 llama.cpp (Dankclaw, retired 2026-08-21) - -A DFlash2 Vulkan deployment briefly replaced CIRU on 2026-08-21, then was -reverted the same day for speed. Its serve script, env, and the build -worktree are archived in `.AGENTTRASH/` and documented in -[serverAppConfigs-docs/LLM/dflash2-large-vulkan.md](serverAppConfigs-docs/LLM/dflash2-large-vulkan.md). - -### FastMTP patched llama.cpp (Dankserver, retired 2026-08-18) - -Historical: the Qwen3.8-27B IQ4_XS FastMTP service was retired on 2026-08-18 -(files in `.AGENTTRASH/`). The build notes are kept for reference. The -HauhauCS FastMTP draft sidecar required a llama.cpp build from commit -`4df29be4f4c3673f428170fda944a5b19f743bb8` plus the -`HauhauCS-FastMTP-llama.cpp.patch` (from the HF repo). The build lives at -`/home/dank/llama.cpp-fastmtp/build` on Dankserver. Loading the draft with a -stock binary fails with `expected 5120, 248320, got 5120, 32768`. Do not -rebuild from another commit without re-applying the patch. - -**REQUIRED build flags:** `-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON`. -Without `GGML_CUDA_FA_ALL_QUANTS` the q8_0 KV cache + flash attention falls -back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s -vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README -for the base recipe. - -### Env var naming rule — HIP runtime collision - -The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`, -and `GPU_MAX_HEAP_SIZE`. The serve scripts export the env file with -`set -a`. A tunable named `TEMP` leaks into HIP. On 2026-08-13 `TEMP=1.0` -made HIP use the path `1.0` for its JIT kernel cache, then segfaulted every -model load. Commit `227ce01` fixed it by renaming the sampling variable to -`TEMPERATURE`. - -Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. Check for -collisions with this command: - -``` -strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE" -``` - -### Crash-loop diagnostic procedure - -When a llama service segfaults during model load, follow these steps. - -```mermaid -graph TD - A[Service SEGVs at load] --> B[Read journal + kernel segfault line] - B --> C{Segfault in libamdhip64?} - C -- yes --> D[Bisect script vs raw binary] - C -- no --> E[Check config, memory, model files] - D --> F[Run gdb for backtrace] - F --> G[Find root cause] -``` - -1. Read the service journal with `journalctl -u -n 100`. Then read - the kernel segfault line with `journalctl -k`. A segfault at address 8 in - `libamdhip64` means a NULL unref in the HIP runtime. -2. Run the serve script directly. Then run the raw binary with the same - flags. A different outcome isolates the env file as the cause. -3. Run gdb for the backtrace with this command: - ``` - gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-medium.sh - ``` - `llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost. - Trust gdb, not the log. -4. Kernel spam about `amdgpu: SVM mapping failed` is a symptom, not the root - cause. Investigate other causes before assuming memory pressure. - -### Operational quirks - -- Both hosts allow passwordless sudo for the deploy user (`(ALL) NOPASSWD: - ALL`, verified 2026-08-22). Use plain `sudo` in committed scripts. Do not - use `echo password | sudo -S` in committed scripts. The `FRANK_SUDO_PW` - env var covers headless runs on a host without that. -- Never run `pkill -f` with a pattern that appears in your own command line. - It kills your ssh session. Use `pkill -x `. -- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM - (`--no-mmap`, MoE experts on CPU) in a few minutes cold. Dankclaw loads the - ~30 GiB Q8_0 AGENT payload (main + MTP draft + mmproj) a few minutes cold - with a fresh KV cache; ~18 s with a warm page cache. Health returns 503 while loading. Do not kill the - process. `deploy.sh` waits up to 20 minutes on both targets (see - `LLM_WAIT`). -- A client abort and a server release look different in the journal. A - client abort logs `W srv stop: cancel task, id_task = N`. A natural stop - (EOS or limit) logs only `slot release` with `truncated = 0`. The server - never preempts a running generation. When all slots are busy, new tasks - defer into a queue. See the 2026-08-13 incident in - [serverAppConfigs-docs/LLM/deployment.md](serverAppConfigs-docs/LLM/deployment.md). -- ComfyUI on Dankclaw runs CPU-only (`--cpu`). ROCm wheels do not support - gfx1151. Do not use CUDA images. -- SearXNG outbound traffic must stay on the Gluetun tunnel. Do not move - its ports off the gluetun service. -- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash - premature-EOS bug are historical. See - [serverAppConfigs-docs/LLM/dankserver-vram-growth.md](serverAppConfigs-docs/LLM/dankserver-vram-growth.md) - and [serverAppConfigs-docs/LLM/qwen3.8-spec-matrix.md](serverAppConfigs-docs/LLM/qwen3.8-spec-matrix.md). - The FastFlowLM NPU endpoint is retired. See - [serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md](serverAppConfigs-docs/LLM/FastFlowLM%20NPU%20Inference.md). - -### Dankclaw memory model - -The GPU budget on Dankclaw is the rocm-smi figure: 64 GiB VRAM reported by -rocm-smi (SVM-backed). The docs' older 96 GiB figure was the addressable -SVM pool. The GPU node reports `local_mem_size 0`, so all allocations are -GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use `free` or -`dmidecode` for the GPU budget. They show the CPU-side carve-out. This is -intentional. Do not change it. - -Host swap on Dankclaw is a **64 GiB** file at `/swap.img` (raised from 8 GiB -on 2026-09-12), fstab-driven via the generated `swap.img.swap` unit. It is -host-level state with no repo file, so it is invisible to `deploy.sh` — check -`swapon --show` before reasoning about headroom. It protects the co-tenants, -not frank-large: the ~68 GiB of pinned weights are not helped by swap, and -swapping the KV pool would be worse than the OOM it prevents. If you ever -resize it, **create and `swapon` the replacement before `swapoff` of the old -one** — otherwise the live pages must land in RAM, and `MemAvailable` overstates -what is there by the size of the model. Full procedure and caveats: -[[Swap Configuration]] (`serverAppConfigs-docs/Dankclaw/Swap Configuration.md`). +Retired runbooks are under `serverAppConfigs-docs/LLM/`. Retired scripts live +in `old/` and `.AGENTTRASH/`. ## Why -Direct edits on the remote drift from version control. Future agents will not -know what changed. Git is the single source of truth. The two workloads share -two hosts, one Caddy, and one Gitea. One repo keeps them in one history. - -> User request → Fetch from remote → Commit here → Edit → Test → Commit → Push → Pull on remote → Dispatch changed service(s) → Health +Direct edits on the remote drift from version control, and future agents +won't know what changed. Git is the single source of truth. The two workloads +share two hosts, one Caddy, and one Gitea; one repo keeps them in one +history. ## Summary | Context | Action | |---------|--------| -| Editing a compose/Caddyfile/serve script | **Work through git** — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote directly. | -| Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before making changes. | -| Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. | -| Deploying a committed change | **Run `scripts/deploy.sh `** — it pushes, pulls with fast-forward-only, plans changed paths, dispatches selected services, and checks their health. See [serverAppConfigs-docs/LLM/deployment.md](serverAppConfigs-docs/LLM/deployment.md). | | Container without a Quadlet unit | **Give it one** — self-healing, boot-persistent `.container` unit via git (see Boot Persistence & Self-Healing). | -| Benchmarking | **Stop the service first**, restart, verify health. See the Benchmarking Policy above. | +| Editing a compose/Caddyfile/serve script | **Through git** — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote. | +| Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before editing. | +| Unstaged local changes | **Commit or discard** — check `git status` before starting work; ask if unsure. | +| Deploying a committed change | **`scripts/deploy.sh `** — pushes, pulls ff-only, plans, dispatches, health-checks. | +| Container without a Quadlet unit | **Give it one** — boot-persistent, self-healing `.container` via git. | +| Benchmarking | **Stop and mask the service first**, bench, unmask, restart, verify health. | | Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. | | Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. | -| Real secrets | **Never commit** — use host `.env` files. `.env.example` holds placeholders only. `FRANK_SUDO_PW` covers headless sudo. | +| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |