Frank
048e7eaaba
User request 2026-08-19 (second pass): parallel slots degraded performance. PARALLEL=1, CTX_SIZE=262144 (native max, -c equals the per-slot context with one slot). Concurrent requests queue on the single slot; the server never preempts a running generation. pi models.json contextWindow/maxTokens back to 262144 (outside repo). Docs updated to match.
223 lines
11 KiB
Markdown
223 lines
11 KiB
Markdown
# AGENTS.md — vllmScripts Agent Rules
|
|
|
|
## Overview
|
|
|
|
This project manages deployment scripts for the LLM inference fleet on two
|
|
local machines. Topology (2026-08-18):
|
|
|
|
| Target | Host | Runtime | Service | Model | Port |
|
|
|--------|------|---------|---------|-------|------|
|
|
| Dankserver | dank@dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
|
|
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView, 262K context / 1 slot | 52625 |
|
|
|
|
The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
|
|
NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw; the
|
|
CIRU ROCmFPX llama.cpp runtime now runs on the Strix Halo iGPU (gfx1151).
|
|
A second 2026-08-18 swap renamed the services and restored the Qwen3.6 35B
|
|
A3B Q6_K_P model on Dankserver: Dankserver is now `frank-default-small`
|
|
(Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE experts on CPU) and Dankclaw is
|
|
now `frank-default-large` (CIRU Qwen3.8-27B, 262K context / 1 slot). The
|
|
retired Qwen3.8-27B IQ4_XS FastMTP files live in `.AGENTTRASH/`. See
|
|
docs/ciru-activefpx-promptforge.md. The old scripts live in `old/scripts/`
|
|
for reference; they describe the retired PP topology.
|
|
|
|
**Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the
|
|
Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its
|
|
OpenWebUI consumes the `:7770` endpoint on Dankserver; its Gitea instance
|
|
hosts this repository.
|
|
|
|
## Benchmarking Policy (MANDATORY)
|
|
|
|
**STOP the production service BEFORE running any benchmark on a target.**
|
|
These servers serve real users (about 3 people). Running a bench while the
|
|
service is up starves the GPU and corrupts results. Restart the service
|
|
when the bench finishes.
|
|
|
|
Rules:
|
|
|
|
1. Stop the service before any benchmark:
|
|
`sudo systemctl stop frank-default-small` (Dankserver) or
|
|
`sudo systemctl stop frank-default-large` (Dankclaw). Use the password
|
|
`Assface.3241` with `sudo -S` when no tty is available.
|
|
2. Run the bench on a scratch port (never the service ports 7770/52625).
|
|
3. Restart the service when the bench completes:
|
|
`sudo systemctl start <svc>`.
|
|
4. Verify health before reporting:
|
|
`curl http://192.168.0.235:7770/health` or
|
|
`curl http://192.168.0.143:52625/health`.
|
|
|
|
## Git Workflow
|
|
|
|
All script edits must flow through git. Do not edit files on the remote machine directly.
|
|
|
|
### Protocol
|
|
|
|
1. **Source the script from the target machine** — fetch it from the server where it runs
|
|
2. **Place it in this repo** — commit the fetched version
|
|
3. **Edit here** — make all changes locally in the working tree
|
|
4. **Commit and push** — push to origin (Gitea on dankclaw)
|
|
5. **Pull on the target** — `git pull` on the remote machine before restarting
|
|
6. **Restart the service** — use the updated script on the target
|
|
|
|
## Deployment & Update Process
|
|
|
|
Use `scripts/deploy.sh` to deploy committed changes to a target. It pushes `main`, pulls on the target, restarts the service, and checks the health endpoint.
|
|
|
|
```
|
|
FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver # llama.cpp FastMTP, port 7770
|
|
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw # CIRU llama.cpp, port 52625
|
|
./scripts/deploy.sh dankserver --dry-run # preview without changes
|
|
```
|
|
|
|
Set `FRANK_SUDO_PW` for headless runs (remote sudo reads it from stdin).
|
|
The passwordless restart helpers installed for the retired topology are
|
|
obsolete; `deploy.sh` no longer uses them.
|
|
|
|
Deployment workflow:
|
|
|
|
```mermaid
|
|
graph LR
|
|
A[Edit locally] --> B[Commit]
|
|
B --> C[Push to origin]
|
|
C --> D[deploy.sh: pull on target]
|
|
D --> E[restart service]
|
|
E --> F[health check]
|
|
```
|
|
|
|
## Pi Model Config (`~/.pi/agent/models.json`)
|
|
|
|
The Pi agent reaches the fleet through `/home/dank/.pi/agent/models.json`.
|
|
This file lives outside the repository. Each provider is one host endpoint.
|
|
Each model `id` is the served alias. The two aliases are
|
|
`frank-default-small` (dankserver `:7770`) and `frank-default-large`
|
|
(dankclaw `:52625`).
|
|
|
|
The model `id` values are stable. The file
|
|
`/home/dank/.pi/agent/settings.json` references them. Two resolution rules
|
|
apply (verified in `@earendil-works/pi-coding-agent` `model-resolver.js`):
|
|
|
|
- The default model resolves as a `defaultProvider` + `defaultModel` pair.
|
|
The `defaultProvider` must name the provider that serves the `defaultModel`.
|
|
- A subagent override uses a bare model `id`. A bare `id` resolves only when
|
|
it is unique across all providers.
|
|
|
|
When you rename a service or move a model to a new host, update
|
|
`models.json`. Update the provider key, the model `id`, `contextWindow`, and
|
|
`input`. If the default model moved hosts, also update `defaultProvider` in
|
|
`settings.json`. Set `contextWindow` to the per-slot context (dankserver
|
|
`175104`, dankclaw `262144`). The full schema is in the agent knowledge base
|
|
page `~/.pi/agent/docs/pi/models.md`.
|
|
|
|
## Runtime & Troubleshooting
|
|
|
|
### FastMTP patched llama.cpp (retired, 2026-08-18)
|
|
|
|
Historical: the Qwen3.8-27B IQ4_XS FastMTP service was retired on 2026-08-18
|
|
(files in `.AGENTTRASH/`). The build notes below are kept for reference.
|
|
The HauhauCS FastMTP draft sidecar required a llama.cpp build from commit
|
|
`4df29be4f4c3673f428170fda944a5b19f743bb8` plus the
|
|
`HauhauCS-FastMTP-llama.cpp.patch` (from the HF repo). The build lives at
|
|
`/home/dank/llama.cpp-fastmtp/build` on Dankserver. Loading the draft with
|
|
a stock binary fails with `expected 5120, 248320, got 5120, 32768`. Do not
|
|
rebuild from another commit without re-applying the patch.
|
|
|
|
**REQUIRED build flags:** `-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON`.
|
|
Without `GGML_CUDA_FA_ALL_QUANTS` the q8_0 KV cache + flash attention falls
|
|
back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s
|
|
vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README
|
|
for the base recipe.
|
|
|
|
### CIRU ROCmFPX llama.cpp (Dankclaw)
|
|
|
|
The CIRU llama-server serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge`
|
|
from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
|
|
The runtime is the ciru-ai/ROCmFPX tag `qwen3.8-activefpx-promptforge-v2.3`
|
|
(commit `103af8b41`) built against the private TheRock ROCm 7.15 SDK kept
|
|
in the retired ling3-ciru venv
|
|
(`.venv/lib/python3.12/site-packages/_rocm_sdk_*`). The model card requires
|
|
ROCm 7.15 for the advertised Strix Halo performance; the system ROCm 7.2.4
|
|
selects slower kernels. The serve script exports the 7.15 SDK libraries
|
|
first in `LD_LIBRARY_PATH`.
|
|
|
|
The model files (v2 GGUF + FFN.pfs + GDN.pfs, ~34.9 GiB) live in
|
|
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/`. The full build runbook is
|
|
in docs/ciru-activefpx-promptforge.md. The retired vLLM venvs
|
|
(`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to
|
|
`/home/dank/.AGENTTRASH/` on 2026-08-18.
|
|
|
|
**Env var naming rule still applies:** the HIP runtime reads `TEMP`, `HOST`,
|
|
and `GPU_MAX_HEAP_SIZE`. The serve script passes `--host`/`--port`/`--temp`
|
|
as CLI flags only.
|
|
|
|
### Env var naming rule — HIP runtime collision
|
|
|
|
The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`,
|
|
and `GPU_MAX_HEAP_SIZE`. The serve scripts export the env file with
|
|
`set -a`. A tunable named `TEMP` leaks into HIP. On 2026-08-13
|
|
`TEMP=1.0` made HIP use the path `1.0` for its JIT kernel cache, then
|
|
segfaulted every model load. Commit `227ce01` fixed it by renaming the
|
|
sampling variable to `TEMPERATURE`.
|
|
|
|
Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. Check for
|
|
collisions with this command:
|
|
|
|
```
|
|
strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"
|
|
```
|
|
|
|
### Crash-loop diagnostic procedure
|
|
|
|
When a llama service segfaults during model load, follow these steps.
|
|
|
|
```mermaid
|
|
graph TD
|
|
A[Service SEGVs at load] --> B[Read journal + kernel segfault line]
|
|
B --> C{Segfault in libamdhip64?}
|
|
C -- yes --> D[Bisect script vs raw binary]
|
|
C -- no --> E[Check config, memory, model files]
|
|
D --> F[Run gdb for backtrace]
|
|
F --> G[Find root cause]
|
|
```
|
|
|
|
1. Read the service journal with `journalctl -u <service> -n 100`. Then read the kernel segfault line with `journalctl -k`. A segfault at address 8 in `libamdhip64` means a NULL unref in the HIP runtime.
|
|
2. Run the serve script directly. Then run the raw binary with the same flags. A different outcome isolates the env file sourcing.
|
|
3. Run gdb for the backtrace with this command:
|
|
```
|
|
gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-large.sh
|
|
```
|
|
`llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost. Trust gdb, not the log.
|
|
4. Kernel spam about `amdgpu: SVM mapping failed` is a symptom, not the root cause. Investigate other causes before assuming memory pressure.
|
|
|
|
### Operational quirks
|
|
|
|
- `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
|
|
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
|
|
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw CIRU loads the 34.9 GiB ActiveFPX payload in ~5-10 minutes. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`).
|
|
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
|
|
- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md).
|
|
|
|
### Dankclaw memory model
|
|
|
|
The GPU budget on Dankclaw is the rocm-smi figure: 64 GiB VRAM reported
|
|
by rocm-smi (SVM-backed; the docs' older 96 GiB figure was the addressable
|
|
SVM pool). The GPU node reports `local_mem_size 0`, so all allocations are
|
|
GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use `free` or
|
|
`dmidecode` for the GPU budget; they show the CPU-side carve-out. This is
|
|
intentional. Do not change it.
|
|
|
|
## Why
|
|
|
|
Direct edits on the remote drift from version control. Future agents will not know what changed. Git is the single source of truth.
|
|
|
|
> User request → Fetch script from remote → Commit here → Edit → Commit → Push → Pull on remote → Restart
|
|
|
|
## Summary
|
|
|
|
| Context | Action |
|
|
|---------|--------|
|
|
| Editing a deployment script | **Work through git** — fetch, commit, edit, push, pull, restart. Never edit on the remote directly. |
|
|
| Script exists only on remote | **Pull it into the repo first** — commit the fetched version before making changes. |
|
|
| Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. |
|
|
| Deploying a committed change | **Run `scripts/deploy.sh <target>`** — it pushes, pulls on the target, restarts the service, and checks health. See [docs/deployment.md](docs/deployment.md). |
|
|
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
|