Files

T

Frank 8652116cec docs: document frank-default-large v3 upgrade (Output-K8 runtime, full-W8 views, BF16 vision); update AGENTS/deploy payload sizes

2026-08-21 20:18:57 -07:00

12 KiB

Raw Blame History

AGENTS.md — vllmScripts Agent Rules

Overview

This project manages deployment scripts for the LLM inference fleet on two local machines. Topology (2026-08-21):

Target	Host	Runtime	Service	Model	Port
Dankserver	dank@dankserver	llama.cpp (upstream b10121)	frank-default-small	HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU	7770
Dankclaw	dank@dankclaw	CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv)	frank-default-large	jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy, PromptForge DualView, 262K context / 1 slot	52625

The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw; the CIRU ROCmFPX llama.cpp runtime runs on the Strix Halo iGPU (gfx1151). On 2026-08-21 a DFlash2 Vulkan deployment briefly replaced CIRU, then was reverted the same day for speed; the DFlash2 files live in .AGENTTRASH/. See docs/ciru-activefpx-promptforge.md. The old scripts live in old/scripts/ for reference; they describe the retired PP topology.

Sibling project: ../serverAppConfigs manages the Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its OpenWebUI consumes the :7770 endpoint on Dankserver; its Gitea instance hosts this repository.

Benchmarking Policy (MANDATORY)

STOP the production service BEFORE running any benchmark on a target. These servers serve real users (about 3 people). Running a bench while the service is up starves the GPU and corrupts results. Restart the service when the bench finishes.

Rules:

Stop the service before any benchmark: sudo systemctl stop frank-default-small (Dankserver) or sudo systemctl stop frank-default-large (Dankclaw). Use the password Assface.3241 with sudo -S when no tty is available.
Run the bench on a scratch port (never the service ports 7770/52625).
Restart the service when the bench completes: sudo systemctl start <svc>.
Verify health before reporting: curl http://192.168.0.235:7770/health or curl http://192.168.0.143:52625/health.

Git Workflow

All script edits must flow through git. Do not edit files on the remote machine directly.

Protocol

Source the script from the target machine — fetch it from the server where it runs
Place it in this repo — commit the fetched version
Edit here — make all changes locally in the working tree
Commit and push — push to origin (Gitea on dankclaw)
Pull on the target — git pull on the remote machine before restarting
Restart the service — use the updated script on the target

Deployment & Update Process

Use scripts/deploy.sh to deploy committed changes to a target. It pushes main, pulls on the target, restarts the service, and checks the health endpoint.

FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver   # llama.cpp FastMTP, port 7770
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw     # CIRU llama.cpp, port 52625
./scripts/deploy.sh dankserver --dry-run           # preview without changes

Set FRANK_SUDO_PW for headless runs (remote sudo reads it from stdin). The passwordless restart helpers installed for the retired topology are obsolete; deploy.sh no longer uses them.

Deployment workflow:

graph LR
    A[Edit locally] --> B[Commit]
    B --> C[Push to origin]
    C --> D[deploy.sh: pull on target]
    D --> E[restart service]
    E --> F[health check]

Pi Model Config (`~/.pi/agent/models.json`)

The Pi agent reaches the fleet through /home/dank/.pi/agent/models.json. This file lives outside the repository. Each provider is one host endpoint. Each model id is the served alias. The two aliases are frank-default-small (dankserver :7770) and frank-default-large (dankclaw :52625).

The model id values are stable. The file /home/dank/.pi/agent/settings.json references them. Two resolution rules apply (verified in @earendil-works/pi-coding-agent model-resolver.js):

The default model resolves as a defaultProvider + defaultModel pair. The defaultProvider must name the provider that serves the defaultModel.
A subagent override uses a bare model id. A bare id resolves only when it is unique across all providers.

When you rename a service or move a model to a new host, update models.json. Update the provider key, the model id, contextWindow, and input. If the default model moved hosts, also update defaultProvider in settings.json. Set contextWindow to the per-slot context (dankserver 262144, dankclaw 262144). The full schema is in the agent knowledge base page ~/.pi/agent/docs/pi/models.md.

Runtime & Troubleshooting

FastMTP patched llama.cpp (retired, 2026-08-18)

Historical: the Qwen3.8-27B IQ4_XS FastMTP service was retired on 2026-08-18 (files in .AGENTTRASH/). The build notes below are kept for reference. The HauhauCS FastMTP draft sidecar required a llama.cpp build from commit 4df29be4f4c3673f428170fda944a5b19f743bb8 plus the HauhauCS-FastMTP-llama.cpp.patch (from the HF repo). The build lives at /home/dank/llama.cpp-fastmtp/build on Dankserver. Loading the draft with a stock binary fails with expected 5120, 248320, got 5120, 32768. Do not rebuild from another commit without re-applying the patch.

REQUIRED build flags: -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON. Without GGML_CUDA_FA_ALL_QUANTS the q8_0 KV cache + flash attention falls back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README for the base recipe.

CIRU ROCmFPX llama.cpp (Dankclaw)

The CIRU llama-server serves jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge from /home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server. The runtime is the ciru-ai/ROCmFPX tag qwen3.8-activefpx-promptforge-v2.3 (commit 103af8b41) plus the v3 output-K8 runtime patch (commit af0eac6 on Dankclaw) built against the private TheRock ROCm 7.15 SDK kept in the retired ling3-ciru venv (.venv/lib/python3.12/site-packages/_rocm_sdk_*). The model card requires ROCm 7.15 for the advertised Strix Halo performance; the system ROCm 7.2.4 selects slower kernels. The serve script exports the 7.15 SDK libraries first in LD_LIBRARY_PATH.

The v3 model files (GGUF + FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj, ~36.7 GiB) live in the v3 staging directory /home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/; the v2 artifacts remain in /home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/ for rollback. The full build runbook is in docs/ciru-activefpx-promptforge.md. The retired vLLM venvs (/home/dank/vllm-venv, /home/dank/vllm-vk-venv) were moved to /home/dank/.AGENTTRASH/ on 2026-08-18.

Env var naming rule still applies: the HIP runtime reads TEMP, HOST, and GPU_MAX_HEAP_SIZE. The serve script passes --host/--port/--temp as CLI flags only.

DFlash2 llama.cpp (Dankclaw, retired 2026-08-21)

A DFlash2 Vulkan deployment briefly replaced CIRU on 2026-08-21, then was reverted the same day for speed. Its serve script, env, and the build worktree are archived in .AGENTTRASH/ and documented in docs/dflash2-large-vulkan.md.

Env var naming rule — HIP runtime collision

The ROCm HIP runtime reads the bare environment variables TEMP, HOST, and GPU_MAX_HEAP_SIZE. The serve scripts export the env file with set -a. A tunable named TEMP leaks into HIP. On 2026-08-13 TEMP=1.0 made HIP use the path 1.0 for its JIT kernel cache, then segfaulted every model load. Commit 227ce01 fixed it by renaming the sampling variable to TEMPERATURE.

Never name a tunable TEMP, HOST, or GPU_MAX_HEAP_SIZE. Check for collisions with this command:

strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"

Crash-loop diagnostic procedure

When a llama service segfaults during model load, follow these steps.

graph TD
    A[Service SEGVs at load] --> B[Read journal + kernel segfault line]
    B --> C{Segfault in libamdhip64?}
    C -- yes --> D[Bisect script vs raw binary]
    C -- no --> E[Check config, memory, model files]
    D --> F[Run gdb for backtrace]
    F --> G[Find root cause]

Read the service journal with journalctl -u <service> -n 100. Then read the kernel segfault line with journalctl -k. A segfault at address 8 in libamdhip64 means a NULL unref in the HIP runtime.
Run the serve script directly. Then run the raw binary with the same flags. A different outcome isolates the env file sourcing.
Run gdb for the backtrace with this command:
```
gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-large.sh
```
llama-server buffers its timed stdout. On SIGSEGV the log tail is lost. Trust gdb, not the log.
Kernel spam about amdgpu: SVM mapping failed is a symptom, not the root cause. Investigate other causes before assuming memory pressure.

Operational quirks

sudo on both boxes needs a tty. Remote sudo steps fail non-interactively. Use echo '<password>' | sudo -S <command>.
Never run pkill -f with a pattern that appears in your own command line. It kills your ssh session. Use pkill -x <exact-name>.
Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (--no-mmap, MoE experts on CPU) in a few minutes cold; Dankclaw CIRU loads the v3 ~36.7 GiB payload in ~5-10 minutes. Health returns 503 while loading. Do not kill the process. deploy.sh waits up to 20 minutes on both targets (see HEALTH_WAIT).
A client abort and a server release look different in the journal. A client abort logs W srv stop: cancel task, id_task = N. A natural stop (EOS or limit) logs only slot release with truncated = 0. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in docs/deployment.md.
Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see docs/dankserver-vram-growth.md and docs/qwen3.8-spec-matrix.md. The FastFlowLM NPU endpoint is retired; see docs/FastFlowLM NPU Inference.md.

Dankclaw memory model

The GPU budget on Dankclaw is the rocm-smi figure: 64 GiB VRAM reported by rocm-smi (SVM-backed; the docs' older 96 GiB figure was the addressable SVM pool). The GPU node reports local_mem_size 0, so all allocations are GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use free or dmidecode for the GPU budget; they show the CPU-side carve-out. This is intentional. Do not change it.

Why

Direct edits on the remote drift from version control. Future agents will not know what changed. Git is the single source of truth.

User request → Fetch script from remote → Commit here → Edit → Commit → Push → Pull on remote → Restart

Summary

Context	Action
Editing a deployment script	Work through git — fetch, commit, edit, push, pull, restart. Never edit on the remote directly.
Script exists only on remote	Pull it into the repo first — commit the fetched version before making changes.
Unstaged local changes	Commit or discard — check `git status` before starting work. Ask the user if unsure.
Deploying a committed change	Run `scripts/deploy.sh <target>` — it pushes, pulls on the target, restarts the service, and checks health. See docs/deployment.md.
Tests exist	Run them — verify tests pass after every change. Do not commit failing tests.

12 KiB Raw Blame History