From 99c7255db5778cbbc02c8b6cbc1008035be66681 Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 22 Aug 2026 15:49:44 -0700 Subject: [PATCH] AGENTS.md: merge the vllmScripts directives into one rulebook One repo, one git protocol, one deploy script. Merged from the two former AGENTS.md files: - Overview: both workloads, both hosts, the 2026-08-22 merge note, current LLM fleet topology (frank-default-small :7770 Dankserver, frank-default-large :52625 Dankclaw) - Git workflow + testing (serverAppConfigs protocol, test step kept) - Deployment: the unified deploy.sh (stack + LLM per target) - Secret rules: no passwords in the repo; FRANK_SUDO_PW covers headless sudo (the live sudo password is NOT carried forward) - Deletion & Trash Policy (from the vllmScripts 2026-08-22 commit) - Benchmarking policy (stop the service before any bench) - Pi model config, runtime & troubleshooting (v3 CIRU active, Kairic/DFlash2/FastMTP retired), HIP env naming rule, crash-loop procedure, operational quirks (passwordless sudo verified 2026-08-22), Dankclaw memory model --- AGENTS.md | 327 ++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 303 insertions(+), 24 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 73dd3e9..62a93a9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,68 +2,347 @@ ## Overview -This repository tracks the Podman container stacks for two homelab servers. Dankserver runs the media/library stack (RomM, Copyparty, Calibre-Web). Dankclaw runs the AI/search/development stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus) plus the observability stack. Caddy reverse-proxies everything on both hosts. +This repository is the single source of truth for the two homelab servers. It +tracks both the Podman container stacks and the LLM inference fleet. One repo, +one checkout per host, one deploy script. + +- **Dankserver** (192.168.0.235) — media/library stack (RomM, Copyparty, + Calibre-Web, the *arr stack, Jellyfin) as bare containers, plus the + `frank-default-small` LLM service. +- **Dankclaw** (192.168.0.143) — AI/search/development/observability stack + (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, + Loki, Alloy) plus the `frank-default-large` LLM service. Caddy on Dankclaw + reverse-proxies both hosts. + +**2026-08-22 merge:** the former `vllmScripts` repository (LLM fleet scripts, +units, env tunables, and runbook) is merged into this repo. Its full git +history is preserved (two-parent import commit under the retired `llm/` +prefix, reorganized in follow-up commits). The retired target checkouts +(`/home/dank/vLLM` on Dankserver, `/home/dank/vllmScripts` on Dankclaw) are +archived. All services now run from `/home/dank/serverAppConfigs` on each +host. + +### LLM fleet topology (2026-08-22) + +| Target | Runtime | Service | Model | Port | +|--------|---------|---------|-------|------| +| Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 | +| Dankclaw | CIRU ROCmFPX llama.cpp, PromptForge v3 runtime (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3 (Q8-FFNEQ-LateQ6, full-W8 views, Output-K8 MTP proxy, BF16 vision), 262K context / 1 slot | 52625 | + +History: the 2026-08-17 redo retired the pipeline-parallel split and the +FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on +Dankclaw. On 2026-08-21 a DFlash2 Vulkan deployment briefly replaced CIRU, +then was reverted the same day for speed. On 2026-08-22 the service briefly +swapped to the Kairic Edge IU4 runtime, then was reverted to v3 the same day +(operator decision). Runbooks: [docs/LLM/index.md](docs/LLM/index.md). +Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`. + +## Repository Layout + +- `dankserver/` — compose.yaml + Caddyfile + .env.example for the media stack +- `dankclaw/` — compose.yaml + Caddyfile + .env.example for the AI stack +- `shared/` — homepage, Prometheus, Loki, Alloy, SearXNG configs +- `scripts/` — `deploy.sh` (unified deployer), LLM serve scripts, systemd + units, env tunables, bench harnesses +- `old/` — retired PP/NPU-era LLM scripts (reference only) +- `docs/` — knowledge base (Obsidian vault). `docs/LLM/` is the LLM fleet +- `tests/` — bats test suite ## Git Workflow -All config edits flow through git. Do not edit files on the remote machine directly. +All edits flow through git. Do not edit files on the remote machine directly. ### Protocol -1. **Source the config from the target machine** — fetch the live compose/Caddyfile from the server where it runs -2. **Place it in this repo** — commit the fetched version (redact secrets into `.env`) +1. **Source the config from the target machine** — fetch the live + compose/Caddyfile/serve script from the server where it runs +2. **Place it in this repo** — commit the fetched version (redact secrets + into `.env`) 3. **Edit here** — make all changes locally in the working tree 4. **Run the tests** — `./tests/run.sh` must pass -5. **Commit and push** — push to origin (Gitea on dankclaw) +5. **Commit and push** — push to origin (Gitea on Dankclaw) 6. **Pull on the target** — `git pull` on the remote machine -7. **Restart the stack** — use `scripts/deploy.sh` to pull + restart + healthcheck +7. **Restart** — `scripts/deploy.sh` pulls, restarts the stack and the LLM + service, and checks health ## Deployment -Use `scripts/deploy.sh` to deploy committed changes to a target. +Use `scripts/deploy.sh` to deploy committed changes to a target. One script +covers both workloads per host: the podman stack and the LLM service. ``` -./scripts/deploy.sh dankclaw # Dankclaw stack -./scripts/deploy.sh dankserver # Dankserver stack +./scripts/deploy.sh dankclaw # AI stack + frank-default-large (:52625) +./scripts/deploy.sh dankserver # media stack + frank-default-small (:7770) ./scripts/deploy.sh dankclaw --dry-run # preview without changes ``` +Set `FRANK_SUDO_PW` for headless runs on a host without passwordless sudo +(remote sudo reads it from stdin via `sudo -S`). The value is never stored +in this repo. + +```mermaid +graph LR + A[Edit locally] --> B[Tests pass] + B --> C[Commit] + C --> D[Push to origin] + D --> E[deploy.sh: pull on target] + E --> F[restart podman stack] + F --> G[relink + restart LLM unit] + G --> H[health: homepage + LLM] +``` + ## Testing -Tests are mandatory before commit. `bats` must be installed on the development machine. +Tests are mandatory before commit. `bats` must be installed on the +development machine. ``` ./tests/run.sh ``` -Test seams: compose validity, env consistency (no leaked secrets), Caddyfile validity, docs completeness, script syntax, deployment flow. See [docs/Architecture/Testing Policy.md](docs/Architecture/Testing Policy.md). +Test seams: compose validity, env consistency (no leaked secrets), Caddyfile +validity, quadlet units, docs completeness, script syntax (all of `scripts/`, +including the LLM serve scripts), LLM unit integrity (units point at files in +this repo), LLM secret hygiene, deployment flow. See +[docs/Architecture/Testing Policy.md](docs/Architecture/Testing%20Policy.md). ## Secret Rules -- Never commit real secrets. The live Gitea database password is test-enforced as a leak check. -- `.env` files are git-ignored; commit only `.env.example` with placeholders. -- Do not document passwords in this repo, including sudo passwords. SSH keys are the auth mechanism. +- Never commit real secrets. The live Gitea database password and the live + sudo password are test-enforced as leak checks. +- `.env` files are git-ignored. Commit only `.env.example` with placeholders. + The LLM env tunables in `scripts/` are tracked by exception (`!scripts/*.env`). + They hold paths and numbers, never credentials. +- Do not document passwords in this repo, including sudo passwords. SSH keys + are the auth mechanism. Headless sudo passes the password via the + `FRANK_SUDO_PW` environment variable. The value never enters the repo. + +## Deletion & Trash Policy (MANDATORY) + +Never run `rm` to delete files. The command deletes permanently. Move every +item you delete into trash instead. The trash keeps the deletion recoverable +and reviewable. + +1. In-repo files: move the item to `.AGENTTRASH/` beside its directory. +2. Home and other paths: create a dated staging directory + `~/.AGENTTRASH/YYYYMMDD-topic` and move each item there. +3. List the staged items and their sizes for the user. +4. Wait for explicit user permission before you empty the trash. + +Rules: + +- `rm`, `rm -rf`, and `git clean -f` are forbidden without explicit user + permission. +- `git * --no-verify` is forbidden. +- A moved item stays in the trash until the user approves its purge. +- The trash is a temporary hold, not a long-term archive. + +## Benchmarking Policy (MANDATORY) + +**STOP the production service BEFORE running any benchmark on a target.** +These servers serve real users (about 3 people). Running a bench while the +service is up starves the GPU and corrupts results. Restart the service when +the bench finishes. + +1. Stop the service before any benchmark: + `sudo systemctl stop frank-default-small` (Dankserver) or + `sudo systemctl stop frank-default-large` (Dankclaw). +2. Run the bench on a scratch port (never the service ports 7770/52625). +3. Restart the service when the bench completes: `sudo systemctl start `. +4. Verify health before reporting: + `curl http://192.168.0.235:7770/health` or + `curl http://192.168.0.143:52625/health`. + +## Pi Model Config (`~/.pi/agent/models.json`) + +The Pi agent reaches the fleet through `/home/dank/.pi/agent/models.json`. +This file lives outside the repository. Each provider is one host endpoint. +Each model `id` is the served alias. The two aliases are +`frank-default-small` (Dankserver `:7770`) and `frank-default-large` +(Dankclaw `:52625`). + +The model `id` values are stable. The file +`/home/dank/.pi/agent/settings.json` references them. Two resolution rules +apply (verified in `@earendil-works/pi-coding-agent` `model-resolver.js`): + +- The default model resolves as a `defaultProvider` + `defaultModel` pair. + The `defaultProvider` must name the provider that serves the `defaultModel`. +- A subagent override uses a bare model `id`. A bare `id` resolves only when + it is unique across all providers. + +When you rename a service or move a model to a new host, update +`models.json`. Update the provider key, the model `id`, `contextWindow`, and +`input`. If the default model moved hosts, also update `defaultProvider` in +`settings.json`. Set `contextWindow` to the per-slot context (both hosts +`262144`). The full schema is in the agent knowledge base page +`~/.pi/agent/docs/pi/models.md`. ## Runtime & Troubleshooting -- The LLM endpoints are bare-metal systemd services managed by the sibling `vllmScripts` repository (`../vllmScripts`), not by this repo: llama.cpp PP head `:7770` and FastFlowLM NPU `:52625` on Dankclaw, RPC worker `:50052` on Dankserver. Do not edit their scripts here. -- Model load on the LLM servers takes 3–8 minutes; `503 Loading model` is normal during load. Do not kill the process. -- `sudo` on both boxes needs a tty. Do not use `echo password | sudo -S` in committed scripts; run sudo interactively. -- Never run `pkill -f` with a pattern that appears in your own command line. Use `pkill -x `. -- ComfyUI on Dankclaw runs CPU-only (`--cpu`); ROCm wheels do not support gfx1151. Do not use CUDA images. -- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its ports off the gluetun service. +### v3 CIRU ActiveFPX PromptForge llama.cpp (Dankclaw, ACTIVE) + +The v3 runtime serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3` +from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` +(PromptForge v2.3 + v3 Output-K8 runtime patch). The v3 profile: +`PROMPTFORGE_MODE=m2048_fused_tail1476`, full-W8 companion views, Output-K8 +MTP draft proxy, BF16 mmproj vision, TEMPERATURE 0.7 / TOP_P 0.95, +batch 4096 / ubatch 4096, 16 context checkpoints, reasoning on (deepseek, +budget -1), `--no-mmap`. Five-file payload (~36.7 GiB) in +`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`. Full runbook: +[docs/LLM/ciru-activefpx-promptforge.md](docs/LLM/ciru-activefpx-promptforge.md). + +### Kairic Edge llama.cpp (Dankclaw, RETIRED experiment 2026-08-22) + +The Kairic Edge llama-server serves `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` +(pinned tag `kairic-edge-qwen38-27b-v1`). It was deployed and reverted to v3 +the same day (operator decision). The runtime routes qualified prompt and +verification shapes through the RDNA 3.5 native IU4 matrix instruction. + +**Release gap — CK `config.h`:** the pinned CK commit ships only +`config.h.in`. The release script and guide never generate +`third_party/composable_kernel/include/ck/config.h`. Generate it via a CK +CMake configure and copy it in before building (full runbook in +[docs/LLM/kairic-edge.md](docs/LLM/kairic-edge.md)). The file is untracked +in CK git. A `git clean` removes it. Local rebuilds are byte-identical +(`d7625a0c...`). + +**Target greedy argmax fast path:** the pinned runner sets +`LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1`. The fast path rejects non-greedy +requests (400) and the card's own chat sampling is non-greedy, so the +production attempt ran it at 0. It still rejects reasoning budgets unless +compatibility mode is used. Do not re-deploy Kairic without re-testing chat +first. + +**Rollback target:** the v3 model files remain in +`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/` and the v3 binary in +`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`. +Restored to v3 on 2026-08-22 (commit `c60156f`). + +### DFlash2 llama.cpp (Dankclaw, retired 2026-08-21) + +A DFlash2 Vulkan deployment briefly replaced CIRU on 2026-08-21, then was +reverted the same day for speed. Its serve script, env, and the build +worktree are archived in `.AGENTTRASH/` and documented in +[docs/LLM/dflash2-large-vulkan.md](docs/LLM/dflash2-large-vulkan.md). + +### FastMTP patched llama.cpp (Dankserver, retired 2026-08-18) + +Historical: the Qwen3.8-27B IQ4_XS FastMTP service was retired on 2026-08-18 +(files in `.AGENTTRASH/`). The build notes are kept for reference. The +HauhauCS FastMTP draft sidecar required a llama.cpp build from commit +`4df29be4f4c3673f428170fda944a5b19f743bb8` plus the +`HauhauCS-FastMTP-llama.cpp.patch` (from the HF repo). The build lives at +`/home/dank/llama.cpp-fastmtp/build` on Dankserver. Loading the draft with a +stock binary fails with `expected 5120, 248320, got 5120, 32768`. Do not +rebuild from another commit without re-applying the patch. + +**REQUIRED build flags:** `-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON`. +Without `GGML_CUDA_FA_ALL_QUANTS` the q8_0 KV cache + flash attention falls +back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s +vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README +for the base recipe. + +### Env var naming rule — HIP runtime collision + +The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`, +and `GPU_MAX_HEAP_SIZE`. The serve scripts export the env file with +`set -a`. A tunable named `TEMP` leaks into HIP. On 2026-08-13 `TEMP=1.0` +made HIP use the path `1.0` for its JIT kernel cache, then segfaulted every +model load. Commit `227ce01` fixed it by renaming the sampling variable to +`TEMPERATURE`. + +Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. Check for +collisions with this command: + +``` +strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE" +``` + +### Crash-loop diagnostic procedure + +When a llama service segfaults during model load, follow these steps. + +```mermaid +graph TD + A[Service SEGVs at load] --> B[Read journal + kernel segfault line] + B --> C{Segfault in libamdhip64?} + C -- yes --> D[Bisect script vs raw binary] + C -- no --> E[Check config, memory, model files] + D --> F[Run gdb for backtrace] + F --> G[Find root cause] +``` + +1. Read the service journal with `journalctl -u -n 100`. Then read + the kernel segfault line with `journalctl -k`. A segfault at address 8 in + `libamdhip64` means a NULL unref in the HIP runtime. +2. Run the serve script directly. Then run the raw binary with the same + flags. A different outcome isolates the env file as the cause. +3. Run gdb for the backtrace with this command: + ``` + gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-large.sh + ``` + `llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost. + Trust gdb, not the log. +4. Kernel spam about `amdgpu: SVM mapping failed` is a symptom, not the root + cause. Investigate other causes before assuming memory pressure. + +### Operational quirks + +- Both hosts allow passwordless sudo for the deploy user (`(ALL) NOPASSWD: + ALL`, verified 2026-08-22). Use plain `sudo` in committed scripts. Do not + use `echo password | sudo -S` in committed scripts. The `FRANK_SUDO_PW` + env var covers headless runs on a host without that. +- Never run `pkill -f` with a pattern that appears in your own command line. + It kills your ssh session. Use `pkill -x `. +- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM + (`--no-mmap`, MoE experts on CPU) in a few minutes cold. Dankclaw v3 + loads the ~36.7 GiB five-file payload in ~18 s with a warm page cache, + a few minutes cold. Health returns 503 while loading. Do not kill the + process. `deploy.sh` waits up to 20 minutes on both targets (see + `LLM_WAIT`). +- A client abort and a server release look different in the journal. A + client abort logs `W srv stop: cancel task, id_task = N`. A natural stop + (EOS or limit) logs only `slot release` with `truncated = 0`. The server + never preempts a running generation. When all slots are busy, new tasks + defer into a queue. See the 2026-08-13 incident in + [docs/LLM/deployment.md](docs/LLM/deployment.md). +- ComfyUI on Dankclaw runs CPU-only (`--cpu`). ROCm wheels do not support + gfx1151. Do not use CUDA images. +- SearXNG outbound traffic must stay on the Gluetun tunnel. Do not move + its ports off the gluetun service. +- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash + premature-EOS bug are historical. See + [docs/LLM/dankserver-vram-growth.md](docs/LLM/dankserver-vram-growth.md) + and [docs/LLM/qwen3.8-spec-matrix.md](docs/LLM/qwen3.8-spec-matrix.md). + The FastFlowLM NPU endpoint is retired. See + [docs/LLM/FastFlowLM NPU Inference.md](docs/LLM/FastFlowLM%20NPU%20Inference.md). + +### Dankclaw memory model + +The GPU budget on Dankclaw is the rocm-smi figure: 64 GiB VRAM reported by +rocm-smi (SVM-backed). The docs' older 96 GiB figure was the addressable +SVM pool. The GPU node reports `local_mem_size 0`, so all allocations are +GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use `free` or +`dmidecode` for the GPU budget. They show the CPU-side carve-out. This is +intentional. Do not change it. ## Why -Direct edits on the remote drift from version control. Future agents will not know what changed. Git is the single source of truth. +Direct edits on the remote drift from version control. Future agents will not +know what changed. Git is the single source of truth. The two workloads share +two hosts, one Caddy, and one Gitea. One repo keeps them in one history. + +> User request → Fetch from remote → Commit here → Edit → Test → Commit → Push → Pull on remote → Deploy + restart ## Summary | Context | Action | |---------|--------| -| Editing a compose/Caddyfile | **Work through git** — fetch, commit, edit, test, push, pull, restart. Never edit on the remote directly. | +| Editing a compose/Caddyfile/serve script | **Work through git** — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote directly. | | Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before making changes. | | Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. | -| Deploying a committed change | **Run `scripts/deploy.sh `** — it pulls on the target, restarts the stack, and checks health. | +| Deploying a committed change | **Run `scripts/deploy.sh `** — it pushes, pulls on the target, restarts the stack and the LLM service, and checks both health endpoints. See [docs/LLM/deployment.md](docs/LLM/deployment.md). | +| Benchmarking | **Stop the service first** — scratch port, restart, verify health. See the Benchmarking Policy above. | +| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. | | Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. | -| Real secrets | **Never commit** — use host `.env` files; `.env.example` holds placeholders only. | +| Real secrets | **Never commit** — use host `.env` files. `.env.example` holds placeholders only. `FRANK_SUDO_PW` covers headless sudo. |