dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

AGENTS.md: merge the vllmScripts directives into one rulebook

Browse Source 
One repo, one git protocol, one deploy script. Merged from the two
former AGENTS.md files:
- Overview: both workloads, both hosts, the 2026-08-22 merge note,
  current LLM fleet topology (frank-default-small :7770 Dankserver,
  frank-default-large :52625 Dankclaw)
- Git workflow + testing (serverAppConfigs protocol, test step kept)
- Deployment: the unified deploy.sh (stack + LLM per target)
- Secret rules: no passwords in the repo; FRANK_SUDO_PW covers
  headless sudo (the live sudo password is NOT carried forward)
- Deletion & Trash Policy (from the vllmScripts 2026-08-22 commit)
- Benchmarking policy (stop the service before any bench)
- Pi model config, runtime & troubleshooting (v3 CIRU active,
  Kairic/DFlash2/FastMTP retired), HIP env naming rule, crash-loop
  procedure, operational quirks (passwordless sudo verified
  2026-08-22), Dankclaw memory model
This commit is contained in:
Frank
2026-08-22 15:49:44 -07:00
parent 8001132744
commit 99c7255db5
1 changed files with 303 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+303 -24
View File
@@ -2,68 +2,347 @@
## Overview
This repository tracks the Podman container stacks for two homelab servers. Dankserver runs the media/library stack (RomM, Copyparty, Calibre-Web). Dankclaw runs the AI/search/development stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus) plus the observability stack. Caddy reverse-proxies everything on both hosts.
This repository is the single source of truth for the two homelab servers. It
tracks both the Podman container stacks and the LLM inference fleet. One repo,
one checkout per host, one deploy script.
- **Dankserver** (192.168.0.235) — media/library stack (RomM, Copyparty,
Calibre-Web, the *arr stack, Jellyfin) as bare containers, plus the
`frank-default-small` LLM service.
- **Dankclaw** (192.168.0.143) — AI/search/development/observability stack
(SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus,
Loki, Alloy) plus the `frank-default-large` LLM service. Caddy on Dankclaw
reverse-proxies both hosts.
**2026-08-22 merge:** the former `vllmScripts` repository (LLM fleet scripts,
units, env tunables, and runbook) is merged into this repo. Its full git
history is preserved (two-parent import commit under the retired `llm/`
prefix, reorganized in follow-up commits). The retired target checkouts
(`/home/dank/vLLM` on Dankserver, `/home/dank/vllmScripts` on Dankclaw) are
archived. All services now run from `/home/dank/serverAppConfigs` on each
host.
### LLM fleet topology (2026-08-22)
| Target | Runtime | Service | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
| Dankclaw | CIRU ROCmFPX llama.cpp, PromptForge v3 runtime (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3 (Q8-FFNEQ-LateQ6, full-W8 views, Output-K8 MTP proxy, BF16 vision), 262K context / 1 slot | 52625 |
History: the 2026-08-17 redo retired the pipeline-parallel split and the
FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
Dankclaw. On 2026-08-21 a DFlash2 Vulkan deployment briefly replaced CIRU,
then was reverted the same day for speed. On 2026-08-22 the service briefly
swapped to the Kairic Edge IU4 runtime, then was reverted to v3 the same day
(operator decision). Runbooks: [docs/LLM/index.md](docs/LLM/index.md).
Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`.
## Repository Layout
- `dankserver/` — compose.yaml + Caddyfile + .env.example for the media stack
- `dankclaw/` — compose.yaml + Caddyfile + .env.example for the AI stack
- `shared/` — homepage, Prometheus, Loki, Alloy, SearXNG configs
- `scripts/` — `deploy.sh` (unified deployer), LLM serve scripts, systemd
units, env tunables, bench harnesses
- `old/` — retired PP/NPU-era LLM scripts (reference only)
- `docs/` — knowledge base (Obsidian vault). `docs/LLM/` is the LLM fleet
- `tests/` — bats test suite
## Git Workflow
All config edits flow through git. Do not edit files on the remote machine directly.
All edits flow through git. Do not edit files on the remote machine directly.
### Protocol
1. **Source the config from the target machine** — fetch the live compose/Caddyfile from the server where it runs
2. **Place it in this repo** — commit the fetched version (redact secrets into `.env`)
1. **Source the config from the target machine** — fetch the live
compose/Caddyfile/serve script from the server where it runs
2. **Place it in this repo** — commit the fetched version (redact secrets
into `.env`)
3. **Edit here** — make all changes locally in the working tree
4. **Run the tests** — `./tests/run.sh` must pass
5. **Commit and push** — push to origin (Gitea on dankclaw)
5. **Commit and push** — push to origin (Gitea on Dankclaw)
6. **Pull on the target** — `git pull` on the remote machine
7. **Restart the stack** — use `scripts/deploy.sh` to pull + restart + healthcheck
7. **Restart** — `scripts/deploy.sh` pulls, restarts the stack and the LLM
service, and checks health
## Deployment
Use `scripts/deploy.sh` to deploy committed changes to a target.
Use `scripts/deploy.sh` to deploy committed changes to a target. One script
covers both workloads per host: the podman stack and the LLM service.
```
./scripts/deploy.sh dankclaw # Dankclaw stack
./scripts/deploy.sh dankserver # Dankserver stack
./scripts/deploy.sh dankclaw # AI stack + frank-default-large (:52625)
./scripts/deploy.sh dankserver # media stack + frank-default-small (:7770)
./scripts/deploy.sh dankclaw --dry-run # preview without changes
```
Set `FRANK_SUDO_PW` for headless runs on a host without passwordless sudo
(remote sudo reads it from stdin via `sudo -S`). The value is never stored
in this repo.
```mermaid
graph LR
A[Edit locally] --> B[Tests pass]
B --> C[Commit]
C --> D[Push to origin]
D --> E[deploy.sh: pull on target]
E --> F[restart podman stack]
F --> G[relink + restart LLM unit]
G --> H[health: homepage + LLM]
```
## Testing
Tests are mandatory before commit. `bats` must be installed on the development machine.
Tests are mandatory before commit. `bats` must be installed on the
development machine.
```
./tests/run.sh
```
Test seams: compose validity, env consistency (no leaked secrets), Caddyfile validity, docs completeness, script syntax, deployment flow. See [docs/Architecture/Testing Policy.md](docs/Architecture/Testing Policy.md).
Test seams: compose validity, env consistency (no leaked secrets), Caddyfile
validity, quadlet units, docs completeness, script syntax (all of `scripts/`,
including the LLM serve scripts), LLM unit integrity (units point at files in
this repo), LLM secret hygiene, deployment flow. See
[docs/Architecture/Testing Policy.md](docs/Architecture/Testing%20Policy.md).
## Secret Rules
- Never commit real secrets. The live Gitea database password is test-enforced as a leak check.
- `.env` files are git-ignored; commit only `.env.example` with placeholders.
- Do not document passwords in this repo, including sudo passwords. SSH keys are the auth mechanism.
- Never commit real secrets. The live Gitea database password and the live
sudo password are test-enforced as leak checks.
- `.env` files are git-ignored. Commit only `.env.example` with placeholders.
The LLM env tunables in `scripts/` are tracked by exception (`!scripts/*.env`).
They hold paths and numbers, never credentials.
- Do not document passwords in this repo, including sudo passwords. SSH keys
are the auth mechanism. Headless sudo passes the password via the
`FRANK_SUDO_PW` environment variable. The value never enters the repo.
## Deletion & Trash Policy (MANDATORY)
Never run `rm` to delete files. The command deletes permanently. Move every
item you delete into trash instead. The trash keeps the deletion recoverable
and reviewable.
1. In-repo files: move the item to `.AGENTTRASH/` beside its directory.
2. Home and other paths: create a dated staging directory
`~/.AGENTTRASH/YYYYMMDD-topic` and move each item there.
3. List the staged items and their sizes for the user.
4. Wait for explicit user permission before you empty the trash.
Rules:
- `rm`, `rm -rf`, and `git clean -f` are forbidden without explicit user
permission.
- `git * --no-verify` is forbidden.
- A moved item stays in the trash until the user approves its purge.
- The trash is a temporary hold, not a long-term archive.
## Benchmarking Policy (MANDATORY)
**STOP the production service BEFORE running any benchmark on a target.**
These servers serve real users (about 3 people). Running a bench while the
service is up starves the GPU and corrupts results. Restart the service when
the bench finishes.
1. Stop the service before any benchmark:
`sudo systemctl stop frank-default-small` (Dankserver) or
`sudo systemctl stop frank-default-large` (Dankclaw).
2. Run the bench on a scratch port (never the service ports 7770/52625).
3. Restart the service when the bench completes: `sudo systemctl start <svc>`.
4. Verify health before reporting:
`curl http://192.168.0.235:7770/health` or
`curl http://192.168.0.143:52625/health`.
## Pi Model Config (`~/.pi/agent/models.json`)
The Pi agent reaches the fleet through `/home/dank/.pi/agent/models.json`.
This file lives outside the repository. Each provider is one host endpoint.
Each model `id` is the served alias. The two aliases are
`frank-default-small` (Dankserver `:7770`) and `frank-default-large`
(Dankclaw `:52625`).
The model `id` values are stable. The file
`/home/dank/.pi/agent/settings.json` references them. Two resolution rules
apply (verified in `@earendil-works/pi-coding-agent` `model-resolver.js`):
- The default model resolves as a `defaultProvider` + `defaultModel` pair.
The `defaultProvider` must name the provider that serves the `defaultModel`.
- A subagent override uses a bare model `id`. A bare `id` resolves only when
it is unique across all providers.
When you rename a service or move a model to a new host, update
`models.json`. Update the provider key, the model `id`, `contextWindow`, and
`input`. If the default model moved hosts, also update `defaultProvider` in
`settings.json`. Set `contextWindow` to the per-slot context (both hosts
`262144`). The full schema is in the agent knowledge base page
`~/.pi/agent/docs/pi/models.md`.
## Runtime & Troubleshooting
- The LLM endpoints are bare-metal systemd services managed by the sibling `vllmScripts` repository (`../vllmScripts`), not by this repo: llama.cpp PP head `:7770` and FastFlowLM NPU `:52625` on Dankclaw, RPC worker `:50052` on Dankserver. Do not edit their scripts here.
- Model load on the LLM servers takes 3–8 minutes; `503 Loading model` is normal during load. Do not kill the process.
- `sudo` on both boxes needs a tty. Do not use `echo password | sudo -S` in committed scripts; run sudo interactively.
- Never run `pkill -f` with a pattern that appears in your own command line. Use `pkill -x <exact-name>`.
- ComfyUI on Dankclaw runs CPU-only (`--cpu`); ROCm wheels do not support gfx1151. Do not use CUDA images.
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its ports off the gluetun service.
### v3 CIRU ActiveFPX PromptForge llama.cpp (Dankclaw, ACTIVE)
The v3 runtime serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3`
from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`
(PromptForge v2.3 + v3 Output-K8 runtime patch). The v3 profile:
`PROMPTFORGE_MODE=m2048_fused_tail1476`, full-W8 companion views, Output-K8
MTP draft proxy, BF16 mmproj vision, TEMPERATURE 0.7 / TOP_P 0.95,
batch 4096 / ubatch 4096, 16 context checkpoints, reasoning on (deepseek,
budget -1), `--no-mmap`. Five-file payload (~36.7 GiB) in
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`. Full runbook:
[docs/LLM/ciru-activefpx-promptforge.md](docs/LLM/ciru-activefpx-promptforge.md).
### Kairic Edge llama.cpp (Dankclaw, RETIRED experiment 2026-08-22)
The Kairic Edge llama-server serves `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge`
(pinned tag `kairic-edge-qwen38-27b-v1`). It was deployed and reverted to v3
the same day (operator decision). The runtime routes qualified prompt and
verification shapes through the RDNA 3.5 native IU4 matrix instruction.
**Release gap — CK `config.h`:** the pinned CK commit ships only
`config.h.in`. The release script and guide never generate
`third_party/composable_kernel/include/ck/config.h`. Generate it via a CK
CMake configure and copy it in before building (full runbook in
[docs/LLM/kairic-edge.md](docs/LLM/kairic-edge.md)). The file is untracked
in CK git. A `git clean` removes it. Local rebuilds are byte-identical
(`d7625a0c...`).
**Target greedy argmax fast path:** the pinned runner sets
`LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1`. The fast path rejects non-greedy
requests (400) and the card's own chat sampling is non-greedy, so the
production attempt ran it at 0. It still rejects reasoning budgets unless
compatibility mode is used. Do not re-deploy Kairic without re-testing chat
first.
**Rollback target:** the v3 model files remain in
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/` and the v3 binary in
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
Restored to v3 on 2026-08-22 (commit `c60156f`).
### DFlash2 llama.cpp (Dankclaw, retired 2026-08-21)
A DFlash2 Vulkan deployment briefly replaced CIRU on 2026-08-21, then was
reverted the same day for speed. Its serve script, env, and the build
worktree are archived in `.AGENTTRASH/` and documented in
[docs/LLM/dflash2-large-vulkan.md](docs/LLM/dflash2-large-vulkan.md).
### FastMTP patched llama.cpp (Dankserver, retired 2026-08-18)
Historical: the Qwen3.8-27B IQ4_XS FastMTP service was retired on 2026-08-18
(files in `.AGENTTRASH/`). The build notes are kept for reference. The
HauhauCS FastMTP draft sidecar required a llama.cpp build from commit
`4df29be4f4c3673f428170fda944a5b19f743bb8` plus the
`HauhauCS-FastMTP-llama.cpp.patch` (from the HF repo). The build lives at
`/home/dank/llama.cpp-fastmtp/build` on Dankserver. Loading the draft with a
stock binary fails with `expected 5120, 248320, got 5120, 32768`. Do not
rebuild from another commit without re-applying the patch.
**REQUIRED build flags:** `-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON`.
Without `GGML_CUDA_FA_ALL_QUANTS` the q8_0 KV cache + flash attention falls
back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s
vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README
for the base recipe.
### Env var naming rule — HIP runtime collision
The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`,
and `GPU_MAX_HEAP_SIZE`. The serve scripts export the env file with
`set -a`. A tunable named `TEMP` leaks into HIP. On 2026-08-13 `TEMP=1.0`
made HIP use the path `1.0` for its JIT kernel cache, then segfaulted every
model load. Commit `227ce01` fixed it by renaming the sampling variable to
`TEMPERATURE`.
Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. Check for
collisions with this command:
```
strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"
```
### Crash-loop diagnostic procedure
When a llama service segfaults during model load, follow these steps.
```mermaid
graph TD
A[Service SEGVs at load] --> B[Read journal + kernel segfault line]
B --> C{Segfault in libamdhip64?}
C -- yes --> D[Bisect script vs raw binary]
C -- no --> E[Check config, memory, model files]
D --> F[Run gdb for backtrace]
F --> G[Find root cause]
```
1. Read the service journal with `journalctl -u <service> -n 100`. Then read
the kernel segfault line with `journalctl -k`. A segfault at address 8 in
`libamdhip64` means a NULL unref in the HIP runtime.
2. Run the serve script directly. Then run the raw binary with the same
flags. A different outcome isolates the env file as the cause.
3. Run gdb for the backtrace with this command:
```
gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-large.sh
```
`llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost.
Trust gdb, not the log.
4. Kernel spam about `amdgpu: SVM mapping failed` is a symptom, not the root
cause. Investigate other causes before assuming memory pressure.
### Operational quirks
- Both hosts allow passwordless sudo for the deploy user (`(ALL) NOPASSWD:
ALL`, verified 2026-08-22). Use plain `sudo` in committed scripts. Do not
use `echo password | sudo -S` in committed scripts. The `FRANK_SUDO_PW`
env var covers headless runs on a host without that.
- Never run `pkill -f` with a pattern that appears in your own command line.
It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM
(`--no-mmap`, MoE experts on CPU) in a few minutes cold. Dankclaw v3
loads the ~36.7 GiB five-file payload in ~18 s with a warm page cache,
a few minutes cold. Health returns 503 while loading. Do not kill the
process. `deploy.sh` waits up to 20 minutes on both targets (see
`LLM_WAIT`).
- A client abort and a server release look different in the journal. A
client abort logs `W srv stop: cancel task, id_task = N`. A natural stop
(EOS or limit) logs only `slot release` with `truncated = 0`. The server
never preempts a running generation. When all slots are busy, new tasks
defer into a queue. See the 2026-08-13 incident in
[docs/LLM/deployment.md](docs/LLM/deployment.md).
- ComfyUI on Dankclaw runs CPU-only (`--cpu`). ROCm wheels do not support
gfx1151. Do not use CUDA images.
- SearXNG outbound traffic must stay on the Gluetun tunnel. Do not move
its ports off the gluetun service.
- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash
premature-EOS bug are historical. See
[docs/LLM/dankserver-vram-growth.md](docs/LLM/dankserver-vram-growth.md)
and [docs/LLM/qwen3.8-spec-matrix.md](docs/LLM/qwen3.8-spec-matrix.md).
The FastFlowLM NPU endpoint is retired. See
[docs/LLM/FastFlowLM NPU Inference.md](docs/LLM/FastFlowLM%20NPU%20Inference.md).
### Dankclaw memory model
The GPU budget on Dankclaw is the rocm-smi figure: 64 GiB VRAM reported by
rocm-smi (SVM-backed). The docs' older 96 GiB figure was the addressable
SVM pool. The GPU node reports `local_mem_size 0`, so all allocations are
GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use `free` or
`dmidecode` for the GPU budget. They show the CPU-side carve-out. This is
intentional. Do not change it.
## Why
Direct edits on the remote drift from version control. Future agents will not know what changed. Git is the single source of truth.
Direct edits on the remote drift from version control. Future agents will not
know what changed. Git is the single source of truth. The two workloads share
two hosts, one Caddy, and one Gitea. One repo keeps them in one history.
> User request → Fetch from remote → Commit here → Edit → Test → Commit → Push → Pull on remote → Deploy + restart
## Summary
| Context | Action |
|---------|--------|
| Editing a compose/Caddyfile | **Work through git** — fetch, commit, edit, test, push, pull, restart. Never edit on the remote directly. |
| Editing a compose/Caddyfile/serve script | **Work through git** — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote directly. |
| Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before making changes. |
| Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. |
| Deploying a committed change | **Run `scripts/deploy.sh <target>`** — it pulls on the target, restarts the stack, and checks health. |
| Deploying a committed change | **Run `scripts/deploy.sh <target>`** — it pushes, pulls on the target, restarts the stack and the LLM service, and checks both health endpoints. See [docs/LLM/deployment.md](docs/LLM/deployment.md). |
| Benchmarking | **Stop the service first** — scratch port, restart, verify health. See the Benchmarking Policy above. |
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
| Real secrets | **Never commit** — use host `.env` files; `.env.example` holds placeholders only. |
| Real secrets | **Never commit** — use host `.env` files. `.env.example` holds placeholders only. `FRANK_SUDO_PW` covers headless sudo. |