serverAppConfigs/AGENTS.md

AGENTS.md — vllmScripts Agent Rules

Overview

This project manages deployment scripts for the LLM inference fleet. Dankclaw runs a llama.cpp pipeline-parallel head serving Qwen3.8-27B-Uncensored Q6_K (imatrix): MTP draft (standalone -draft-Q8_0 sidecar, --spec-draft-n-max 4) over the --spec-default ngram-mod fallback, 1 slot at 262144 context (pool 262144), tensors split 0.68/0.32 (45 of 65 blocks) to the Dankserver RTX 3090 ggml-rpc worker, KV q8_0/f16 (mismatched, FA retained via the GGML_CUDA_FA_ALL_QUANTS rebuild), head GPU side on ROCm0 (no --device flag), port 7770. The Dankclaw NPU endpoint (FastFlowLM, frank-default-small, port 52625) serves Qwythos-9B-Claude-Mythos-5-1M (abliterated) as registry tag qwythos:9b at 262144 ctx (swapped in 2026-08-17; previously qwen3.6-moe:35b-a3b; the old llama.cpp 35B service on Dankserver is stopped). Boot policy (2026-08-17): only frank-default-large and frank-pp-worker are enabled for boot. The NPU unit frank-default-small is disabled; the running service keeps serving until stopped or rebooted. See docs/deployment.md for the deployment process, the boot policy, and the re-enable recipes.

Sibling project: ../serverAppConfigs manages the Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its OpenWebUI consumes the :7770 endpoint; its Gitea instance hosts this repository. Cross-links live in both knowledge-base indexes.

Benchmarking Policy (MANDATORY)

STOP the production service BEFORE running any benchmark on a target. These servers serve real users (about 3 people). Running a bench while the service is up starves the GPU and corrupts results. Restart the service when the bench finishes. The users will call if they need the model.

Rules:

1. Stop the service before any benchmark: systemctl stop frank-default-large (Dankclaw) or systemctl stop frank-default-small (Dankclaw NPU). Use the passwordless helper or the password Assface.3241 with sudo -S.
2. Run the bench on a scratch port (never the service port 7770).
3. Restart the service when the bench completes: systemctl start <svc>.
4. Verify health before reporting: curl http://127.0.0.1:7770/health.

Do not skip this policy. A stale GPU contention or a dead production box is worse than a slow bench.

Git Workflow

All script edits must flow through git. Do not edit files on the remote machine directly.

Protocol

1. Source the script from the target machine — fetch it from the server where it runs
2. Place it in this repo — commit the fetched version
3. Edit here — make all changes locally in the working tree
4. Commit and push — push to origin (Gitea on dankclaw)
5. Pull on the target — git pull on the remote machine before restarting
6. Restart the service — use the updated script on the target

Deployment & Update Process

Use scripts/deploy.sh to deploy committed changes to a target. It pushes main, pulls on the target, restarts the service, and checks the health endpoint.

./scripts/deploy.sh dankclaw        # PP head (frank-default-large, port 7770)
./scripts/deploy.sh dankserver      # PP worker (frank-pp-worker, RPC 50052)
./scripts/deploy.sh dankclaw --dry-run   # preview without changes

Deployment workflow:

graph LR
    A[Edit locally] --> B[Commit]
    B --> C[Push to origin]
    C --> D[deploy.sh: pull on target]
    D --> E[restart service]
    E --> F[health check]

Target map:

Target	Host	Runtime	Service	Port
Dankclaw	dank@dankclaw	llama.cpp PP head (Qwen3.8-27B-Uncensored Q6_K, split 0.68/0.32 to the Dankserver worker via RPC, standalone MTP draft + ngram-mod, q8_0/f16 KV)	frank-default-large	7770
Dankclaw	dank@dankclaw	FastFlowLM NPU (XDNA2 50 TOPS, qwythos:9b Q4NX 7.76 GB, 262144 ctx; see FastFlowLM NPU Inference) — disabled for boot since 2026-08-17	frank-default-small	52625
Dankserver	dank@dankserver	ggml-rpc worker (RTX 3090, receives the head's tensor share)	frank-pp-worker	50052

Warning: deploy.sh dankserver is currently broken. It targets the removed frank-default-small unit. Restart the worker with sudo systemctl restart frank-pp-worker until the script is fixed.

Runtime & Troubleshooting

Env var naming rule — HIP runtime collision

The ROCm HIP runtime reads the bare environment variables TEMP, HOST, and GPU_MAX_HEAP_SIZE. The serve scripts export the env file with set -a. A tunable named TEMP leaks into HIP. On 2026-08-13 TEMP=1.0 made HIP use the path 1.0 for its JIT kernel cache, then segfaulted every model load. Commit 227ce01 fixed it by renaming the sampling variable to TEMPERATURE.

Never name a tunable TEMP, HOST, or GPU_MAX_HEAP_SIZE. Check for collisions with this command:

strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"

Crash-loop diagnostic procedure

When a llama service segfaults during model load, follow these steps.

graph TD
    A[Service SEGVs at load] --> B[Read journal + kernel segfault line]
    B --> C{Segfault in libamdhip64?}
    C -- yes --> D[Bisect script vs raw binary]
    C -- no --> E[Check config, memory, model files]
    D --> F[Run gdb for backtrace]
    F --> G[Find root cause]

1. Read the service journal with journalctl -u <service> -n 100. Then read the kernel segfault line with journalctl -k. A segfault at address 8 in libamdhip64 means a NULL unref in the HIP runtime.
2. Run the serve script directly. Then run the raw binary with the same flags. A different outcome isolates the env file sourcing.
3. Run gdb for the backtrace with this command:

   gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-pp-head.sh
   

   llama-server buffers its timed stdout. On SIGSEGV the log tail is lost. Trust gdb, not the log.
4. Kernel spam about amdgpu: SVM mapping failed is a symptom, not the root cause. Investigate other causes before assuming memory pressure.

Operational quirks

• sudo on both boxes needs a tty. The remote sudo steps in deploy.sh fail non-interactively. Use echo '<password>' | sudo -S <command>.
• Passwordless restarts: each target has a root-owned helper and a sudoers NOPASSWD drop-in (/etc/sudoers.d/frank-*). The Dankclaw helper /usr/local/sbin/frank-restart-large.sh is correct for the PP head. The Dankserver helper frank-restart-small.sh is obsolete — it restarts the removed frank-default-small unit. Restart the worker with sudo systemctl restart frank-pp-worker. The sudoers rules match commands exactly: no extra arguments (for example, --no-pager breaks the systemctl status rule). See docs/deployment.md.
• Never run pkill -f with a pattern that appears in your own command line. It kills your ssh session. Use pkill -x <exact-name>.
• Model load is slow: about 3 minutes warm, 8-17 minutes cold (measured 17 minutes on 2026-08-13 after a cache drop). Health returns 503 Loading model the whole time. That is normal. Do not kill the process. deploy.sh waits up to 20 minutes on Dankclaw (see HEALTH_WAIT).
• Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. deploy.sh drops caches on Dankclaw before restart (see DROP_CACHES); manual restarts must run sync && echo 3 | sudo tee /proc/sys/vm/drop_caches first.
• A client abort and a server release look different in the journal. A client abort logs W srv stop: cancel task, id_task = N. A natural stop (EOS or limit) logs only slot release with truncated = 0. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in docs/deployment.md.
• Dankserver crash signature (historical): the retired 35B service aborted under parallel load with CUDA error: an unsupported value or parameter was passed to the function in cublasSgemm_v2 at ggml-cuda.cu:1541, then status=6/ABRT, then 503 Loading model for ~5 minutes (llama.cpp b10121, driver 595.84). Root cause was never determined; the service is retired. Do not confuse that crash with speculative decoding. See docs/dankserver-vram-growth.md.
• DFlash premature EOS under parallel load (historical): on Dankclaw, draft-dflash ended a generating slot ~1s after a new task landed with high prompt-cache reuse (LCP ~0.99), with finish_reason "stop" and no cancel task log — a DFlash shared-draft-context bug. DFlash and DSpark are retired from Dankclaw. Production uses the standalone MTP draft over the ngram-mod fallback at PARALLEL=1 on ROCm0. See docs/qwen3.8-spec-matrix.md.

Dankclaw memory model

The GPU budget on Dankclaw is the rocm-smi figure: 96 GiB addressable VRAM (SVM). The GPU node reports local_mem_size 0, so all allocations are GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use free or dmidecode for the GPU budget; they show the CPU-side carve-out. This is intentional. Do not change it.

Why

Direct edits on the remote drift from version control. Future agents will not know what changed. Git is the single source of truth.

User request → Fetch script from remote → Commit here → Edit → Commit → Push → Pull on remote → Restart

Summary

Context	Action
Editing a deployment script	Work through git — fetch, commit, edit, push, pull, restart. Never edit on the remote directly.
Script exists only on remote	Pull it into the repo first — commit the fetched version before making changes.
Unstaged local changes	Commit or discard — check git status before starting work. Ask the user if unsure.
Deploying a committed change	Run scripts/deploy.sh <target> — it pushes, pulls on the target, restarts the service, and checks health. See docs/deployment.md.
Tests exist	Run them — verify tests pass after every change. Do not commit failing tests.