dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Reorganize: drop llm/ root files (absorbed into repo root in follow-up)

Browse Source 
llm/AGENTS.md, llm/README.md, and llm/.gitignore are superseded by the
merged root AGENTS.md, README.md, and .gitignore.
This commit is contained in:
Frank
2026-08-22 15:07:31 -07:00
parent 1f565c08c3
commit 02f92386f8
3 changed files with 0 additions and 325 deletions
Download Patch File Download Diff File Expand all files Collapse all files
llm/.gitignore
-6
View File
@@ -1,6 +0,0 @@
.pi-subagents/
.AGENTTRASH/
__pycache__/
*.pyc
.venv/
.obsidian/
llm/AGENTS.md
-280
View File
@@ -1,280 +0,0 @@
# AGENTS.md — vllmScripts Agent Rules
## Overview
This project manages deployment scripts for the LLM inference fleet on two
local machines. Topology (2026-08-22):
| Target | Host | Runtime | Service | Model | Port |
|--------|------|---------|---------|-------|------|
| Dankserver | dank@dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp, PromptForge v3 runtime (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3 (Q8-FFNEQ-LateQ6, full-W8 views, Output-K8 MTP proxy, BF16 vision), 262K context / 1 slot | 52625 |
The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw;
the CIRU ROCmFPX runtime then served the v3 ActiveFPX PromptForge model on
the Strix Halo iGPU (gfx1151). On 2026-08-22 the service briefly swapped to
the Kairic Edge IU4 runtime (pinned tag `kairic-edge-qwen38-27b-v1`), then
was reverted to v3 the same day (operator decision); the Kairic build and
model files remain on the machine. On 2026-08-21 a DFlash2 Vulkan
deployment briefly replaced CIRU, then was reverted the same day for
speed; the DFlash2 files live in `.AGENTTRASH/`. See
docs/ciru-activefpx-promptforge.md (active) and docs/kairic-edge.md
(retired experiment). The old scripts
live in `old/scripts/` for reference; they describe the retired PP
topology.
**Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the
Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its
OpenWebUI consumes the `:7770` endpoint on Dankserver; its Gitea instance
hosts this repository.
## Benchmarking Policy (MANDATORY)
**STOP the production service BEFORE running any benchmark on a target.**
These servers serve real users (about 3 people). Running a bench while the
service is up starves the GPU and corrupts results. Restart the service
when the bench finishes.
Rules:
1. Stop the service before any benchmark:
`sudo systemctl stop frank-default-small` (Dankserver) or
`sudo systemctl stop frank-default-large` (Dankclaw). Use the password
`Assface.3241` with `sudo -S` when no tty is available.
2. Run the bench on a scratch port (never the service ports 7770/52625).
3. Restart the service when the bench completes:
`sudo systemctl start <svc>`.
4. Verify health before reporting:
`curl http://192.168.0.235:7770/health` or
`curl http://192.168.0.143:52625/health`.
## Git Workflow
All script edits must flow through git. Do not edit files on the remote machine directly.
### Protocol
1. **Source the script from the target machine** — fetch it from the server where it runs
2. **Place it in this repo** — commit the fetched version
3. **Edit here** — make all changes locally in the working tree
4. **Commit and push** — push to origin (Gitea on dankclaw)
5. **Pull on the target** — `git pull` on the remote machine before restarting
6. **Restart the service** — use the updated script on the target
## Deletion & Trash Policy (MANDATORY)
Never run `rm` to delete files. The command deletes permanently. Move every
item you would delete into `~/.AGENTTRASH/` instead. The trash keeps the
deletion recoverable and reviewable.
Procedure:
1. Create a dated staging directory: `mkdir -p ~/.AGENTTRASH/YYYYMMDD-topic`
2. Move each item: `mv <path> ~/.AGENTTRASH/YYYYMMDD-topic/`
3. List the staged items and their sizes for the user.
4. Wait for explicit user permission before you empty the trash.
Rules:
- `rm`, `rm -rf`, and `git clean -f` are forbidden without explicit user
permission.
- `git * --no-verify` is forbidden.
- A moved item stays in the trash until the user approves its purge.
- The trash is a temporary hold, not a long-term archive.
Use `mv`, never `rm`. This applies to every host, including Dankserver and
Dankclaw.
## Deployment & Update Process
Use `scripts/deploy.sh` to deploy committed changes to a target. It pushes `main`, pulls on the target, restarts the service, and checks the health endpoint.
```
FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver # llama.cpp FastMTP, port 7770
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw # CIRU llama.cpp, port 52625
./scripts/deploy.sh dankserver --dry-run # preview without changes
```
Set `FRANK_SUDO_PW` for headless runs (remote sudo reads it from stdin).
The passwordless restart helpers installed for the retired topology are
obsolete; `deploy.sh` no longer uses them.
Deployment workflow:
```mermaid
graph LR
A[Edit locally] --> B[Commit]
B --> C[Push to origin]
C --> D[deploy.sh: pull on target]
D --> E[restart service]
E --> F[health check]
```
## Pi Model Config (`~/.pi/agent/models.json`)
The Pi agent reaches the fleet through `/home/dank/.pi/agent/models.json`.
This file lives outside the repository. Each provider is one host endpoint.
Each model `id` is the served alias. The two aliases are
`frank-default-small` (dankserver `:7770`) and `frank-default-large`
(dankclaw `:52625`).
The model `id` values are stable. The file
`/home/dank/.pi/agent/settings.json` references them. Two resolution rules
apply (verified in `@earendil-works/pi-coding-agent` `model-resolver.js`):
- The default model resolves as a `defaultProvider` + `defaultModel` pair.
The `defaultProvider` must name the provider that serves the `defaultModel`.
- A subagent override uses a bare model `id`. A bare `id` resolves only when
it is unique across all providers.
When you rename a service or move a model to a new host, update
`models.json`. Update the provider key, the model `id`, `contextWindow`, and
`input`. If the default model moved hosts, also update `defaultProvider` in
`settings.json`. Set `contextWindow` to the per-slot context (dankserver
`262144`, dankclaw `262144`). The full schema is in the agent knowledge base
page `~/.pi/agent/docs/pi/models.md`.
## Runtime & Troubleshooting
### FastMTP patched llama.cpp (retired, 2026-08-18)
Historical: the Qwen3.8-27B IQ4_XS FastMTP service was retired on 2026-08-18
(files in `.AGENTTRASH/`). The build notes below are kept for reference.
The HauhauCS FastMTP draft sidecar required a llama.cpp build from commit
`4df29be4f4c3673f428170fda944a5b19f743bb8` plus the
`HauhauCS-FastMTP-llama.cpp.patch` (from the HF repo). The build lives at
`/home/dank/llama.cpp-fastmtp/build` on Dankserver. Loading the draft with
a stock binary fails with `expected 5120, 248320, got 5120, 32768`. Do not
rebuild from another commit without re-applying the patch.
**REQUIRED build flags:** `-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON`.
Without `GGML_CUDA_FA_ALL_QUANTS` the q8_0 KV cache + flash attention falls
back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s
vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README
for the base recipe.
### v3 CIRU ActiveFPX PromptForge llama.cpp (Dankclaw, ACTIVE)
The v3 runtime serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3`
from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`
(PromptForge v2.3 + v3 Output-K8 runtime patch). The v3 profile:
`PROMPTFORGE_MODE=m2048_fused_tail1476`, full-W8 companion views, Output-K8
MTP draft proxy, BF16 mmproj vision, TEMPERATURE 0.7 / TOP_P 0.95,
batch 4096 / ubatch 4096, 16 context checkpoints, reasoning on (deepseek,
budget -1), `--no-mmap`. Five-file payload (~36.7 GiB) in
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`. Full runbook:
docs/ciru-activefpx-promptforge.md.
### Kairic Edge llama.cpp (Dankclaw, RETIRED experiment 2026-08-22)
The Kairic Edge llama-server serves `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge`
from `/home/dank/ciru-qwen38/ROCmFPX-kairic/build-kairic/bin/llama-server`
(version 79, commit `e97b324`, pinned tag `kairic-edge-qwen38-27b-v1`).
It was deployed and reverted to v3 the same day (operator decision).
The runtime routes qualified prompt and verification shapes through the
RDNA 3.5 native IU4 matrix instruction. The build uses the release script
against the private TheRock ROCm 7.15 SDK kept in the retired ling3-ciru
venv. The model files (GGUF + FFN/GDN/GDN-Output .pfs, 26.05 GiB) live in
`/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge/`.
**Release gap — CK `config.h`:** the pinned CK commit ships only
`config.h.in`; the release script and guide never generate
`third_party/composable_kernel/include/ck/config.h`. Generate it via a CK
CMake configure and copy it in before building (full runbook in
docs/kairic-edge.md). The file is untracked in CK git; a `git clean`
removes it. Local rebuilds are byte-identical (`d7625a0c...`).
**Target greedy argmax fast path:** the pinned runner sets
`LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1`. The fast path rejects non-greedy
requests (400) and the card's own chat sampling is non-greedy, so the
production attempt ran it at 0. It still rejects reasoning budgets unless
compatibility mode is used. Do not re-deploy Kairic without re-testing
chat first.
**Rollback target:** the v3 model files remain in
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/` and the v3 binary in
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
Restored to v3 on 2026-08-22 (commit `c60156f`).
**Env var naming rule still applies:** the HIP runtime reads `TEMP`,
`HOST`, and `GPU_MAX_HEAP_SIZE`. The serve script passes `--host`/`--port`
as CLI flags only; bind and port come from `BIND_ADDR`/`LISTEN_PORT`.
### DFlash2 llama.cpp (Dankclaw, retired 2026-08-21)
A DFlash2 Vulkan deployment briefly replaced CIRU on 2026-08-21, then was
reverted the same day for speed. Its serve script, env, and the build worktree
are archived in `.AGENTTRASH/` and documented in docs/dflash2-large-vulkan.md.
### Env var naming rule — HIP runtime collision
The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`,
and `GPU_MAX_HEAP_SIZE`. The serve scripts export the env file with
`set -a`. A tunable named `TEMP` leaks into HIP. On 2026-08-13
`TEMP=1.0` made HIP use the path `1.0` for its JIT kernel cache, then
segfaulted every model load. Commit `227ce01` fixed it by renaming the
sampling variable to `TEMPERATURE`.
Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. Check for
collisions with this command:
```
strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"
```
### Crash-loop diagnostic procedure
When a llama service segfaults during model load, follow these steps.
```mermaid
graph TD
A[Service SEGVs at load] --> B[Read journal + kernel segfault line]
B --> C{Segfault in libamdhip64?}
C -- yes --> D[Bisect script vs raw binary]
C -- no --> E[Check config, memory, model files]
D --> F[Run gdb for backtrace]
F --> G[Find root cause]
```
1. Read the service journal with `journalctl -u <service> -n 100`. Then read the kernel segfault line with `journalctl -k`. A segfault at address 8 in `libamdhip64` means a NULL unref in the HIP runtime.
2. Run the serve script directly. Then run the raw binary with the same flags. A different outcome isolates the env file sourcing.
3. Run gdb for the backtrace with this command:
```
gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-large.sh
```
`llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost. Trust gdb, not the log.
4. Kernel spam about `amdgpu: SVM mapping failed` is a symptom, not the root cause. Investigate other causes before assuming memory pressure.
### Operational quirks
- `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw v3 loads the ~36.7 GiB five-file payload (GGUF + FFN/GDN + Output-K8 .pfs + mmproj) in ~18 s with a warm page cache, a few minutes cold. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`).
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md).
### Dankclaw memory model
The GPU budget on Dankclaw is the rocm-smi figure: 64 GiB VRAM reported
by rocm-smi (SVM-backed; the docs' older 96 GiB figure was the addressable
SVM pool). The GPU node reports `local_mem_size 0`, so all allocations are
GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use `free` or
`dmidecode` for the GPU budget; they show the CPU-side carve-out. This is
intentional. Do not change it.
## Why
Direct edits on the remote drift from version control. Future agents will not know what changed. Git is the single source of truth.
> User request → Fetch script from remote → Commit here → Edit → Commit → Push → Pull on remote → Restart
## Summary
| Context | Action |
|---------|--------|
| Editing a deployment script | **Work through git** — fetch, commit, edit, push, pull, restart. Never edit on the remote directly. |
| Script exists only on remote | **Pull it into the repo first** — commit the fetched version before making changes. |
| Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. |
| Deploying a committed change | **Run `scripts/deploy.sh <target>`** — it pushes, pulls on the target, restarts the service, and checks health. See [docs/deployment.md](docs/deployment.md). |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
llm/README.md
-39
View File
@@ -1,39 +0,0 @@
# vLLM Scripts
Deployment scripts for the LLM inference fleet (2026-08-18 topology):
upstream llama.cpp on Dankserver and CIRU ROCmFPX llama.cpp on Dankclaw.
## Targets
| Host | User | Codename | GPU |
|------|------|----------|-----|
| dankserver | dank | Dankserver | NVIDIA RTX 3090 24 GB + 62 GB RAM |
| dankclaw | dank | Dankclaw | AMD Strix Halo iGPU gfx1151 (unified memory) |
## Active Services
| Service | Host | Runtime | Model | Port |
|---------|------|---------|-------|------|
| frank-default-small | dankserver | llama.cpp (upstream b10121) | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU, 4 slots x ~175K ctx | 7770 |
| frank-default-large | dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK) | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 262K ctx / 1 slot | 52625 |
The 2026-08-17 redo retired the pipeline-parallel split (PP head + RPC
worker) and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the
vLLM service on Dankclaw and renamed the services (see AGENTS.md). Their
scripts are preserved in `old/scripts/` for reference.
## Structure
- `docs/` — knowledge base: deployment runbook, model notes, research archive
- `scripts/` — serve scripts, env files, systemd units, deploy and test tools
- `old/scripts/` — retired PP/NPU-era scripts (reference only)
## Deployment
See [docs/deployment.md](docs/deployment.md). Run
`FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver` for the llama.cpp box or
`FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw` for the CIRU box.
## Knowledge Base
See [docs/index.md](docs/index.md) for the project knowledge base index.