serverAppConfigs/docs/deployment.md

Deployment and Update Process

This page describes how to deploy and update the LLM servers on Dankserver and Dankclaw. The process uses git as the single source of truth. All script edits flow through this repository.

Overview

Two servers run LLM inference services. Each server pulls its scripts from this repository. The deployment process moves a committed change from the repository to a running service.

Target	Host	Runtime	Service	Port
Dankclaw	dank@dankclaw	vLLM-ROCm (CIRU INT4)	frank-default-large	7770
Dankserver	dank@dankserver	llama.cpp (Q6_K_P + ngram-mod, MoE offload)	frank-default-small	7770

Workflow

The deployment workflow has six steps.

graph LR
    A[Edit scripts locally] --> B[Commit]
    B --> C[Push to origin<br>Gitea on Dankclaw]
    C --> D[Pull on target<br>git pull]
    D --> E[Restart systemd service]
    E --> F[Verify health endpoint]

1. Fetch the current script from the target machine.
2. Commit the fetched version in this repository.
3. Edit the script locally in the working tree.
4. Commit and push the change to origin.
5. Pull the change on the target machine.
6. Restart the service and verify that it is healthy.

Deploy Script

The scripts/deploy.sh script executes steps 4 through 6. Run it from the repository root.

Run this command to deploy to Dankclaw:

./scripts/deploy.sh dankclaw

Run this command to deploy to Dankserver:

./scripts/deploy.sh dankserver

Add the --dry-run flag to preview without making changes:

./scripts/deploy.sh dankclaw --dry-run

The script does these actions:

1. Verifies that the local working tree has no uncommitted changes.
2. Pushes the main branch to origin.
3. Pulls the repository on the target machine.
4. Installs or updates the systemd unit.
5. Drops page caches on Dankclaw before the restart. This prevents the load stall that happens when the model's pages are fully cached.
6. Reloads systemd and restarts the service.
7. Waits for the health endpoint to respond (over ssh).

Set FRANK_SUDO_PW when running headless, so that remote sudo reads the password from stdin:

FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw

Update Dankclaw

1. Pull the repository.
2. Symlink the unit file, if it changed.
3. Drop the page caches.
4. Reload and restart the service.

Run these commands on Dankclaw:

cd /home/dank/vllmScripts
git pull origin main
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
sudo systemctl daemon-reload
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:7770/health

Update Dankserver

Run these commands on Dankserver:

cd /home/dank/vLLM
git pull origin main
sudo ln -sf /home/dank/vLLM/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-small
curl -sf http://127.0.0.1:7770/health

Service Units

Dankclaw — frank-default-large

The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B uncensored Q8_K_P model under the alias frank-default-large on port 7770. It uses a DFlash draft model (draft-dflash) for speculative decoding on the ROCm (HIP) backend, with 3 parallel slots at 200700 context each (pool 602100), KV cache q8_0 K / q8_0 V on GPU (pairing required for Flash Attention on this build), full layer offload (--gpu-layers all, --fit off), and batch 2048 / ubatch 512. The unit file is scripts/frank-default-large.service. It launches through scripts/serve-llama-frank-large.sh, which reads tunables from scripts/frank-default-large.env.

Note: this service replaced the retired vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime and the earlier llama-serve/vLLM units on Dankclaw. See frank-default-large-benchmark for the benchmark that drove the backend and spec choices.

Dankserver — frank-default-small

The active service on Dankserver runs llama.cpp (upstream, b10108+) and serves the HauhauCS Qwen3.6-35B-A3B uncensored Q6_K_P model under the alias frank-default-small on port 7770. It uses ngram-mod speculative decoding (--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24), 4 parallel slots at 200000 context each (pool 800000), and mmproj vision. 35 of 40 MoE expert layers stay on CPU (--n-cpu-moe 35); 5 expert layers' weights are back on GPU alongside all non-expert tensors (--gpu-layers all). KV stays on GPU (K=q8_0, V=f16). Batch 8192 / ubatch 2048, --ctx-checkpoints 4, --no-mmap. The unit file is scripts/frank-default-small.service, symlinked to /etc/systemd/system/. It launches through scripts/serve-llama-frank-small.sh, which reads tunables from scripts/frank-default-small.env.

Note: this service replaced the retired vllm-awq-qwen36 (AWQ 4-bit, port 34401) and the earlier vllm-qwen36 (FP8) / llama-server (ROCmFPX) units. The repo tracks main on Dankserver at /home/dank/vLLM.

Validation

The deploy script checks the health endpoint after a restart. For deeper checks, run the validation script on the target.

Run this command on Dankserver to test vLLM throughput:

./scripts/validate-vllm.sh

Troubleshooting

If the health check fails after a deploy:

1. Read the service logs with journalctl -u <service> -n 200.
2. Confirm that the model files exist in the cache.
3. Confirm that the GPU has free memory.
4. Check the service status with systemctl status <service>.

Incident: HIP Runtime Crash From the TEMP Env Variable (2026-08-13)

On 2026-08-13 the frank-default-large service crash-looped for hours. Every restart died with SIGSEGV about 20 seconds into model load, at a fixed offset in libamdhip64 (NULL refcount unref during ggml_backend_cuda_buffer_set_tensor). The kernel log showed bursts of amdgpu: SVM mapping failed, exceeds resident system memory limit, which pointed at memory but was a red herring.

Root cause: the env files exported TEMP=1.0 (a sampling default added in commit db3eb01). The ROCm HIP runtime reads the TEMP environment variable as its JIT kernel-cache directory. HIP then used the path 1.0, failed, and segfaulted on a NULL object. The crash was deterministic whenever the serve script sourced the env file, and absent when it did not.

Fix: rename the sampling variable to TEMPERATURE in the env files and serve scripts (commit 227ce01). The serve script still passes --temp "$TEMPERATURE" to llama-server, so sampling defaults are unchanged.

If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Check with strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|...". Do not name any tunable TEMP, HOST, or GPU_MAX_HEAP_SIZE.

See Also

• AGENTS.md — the git workflow rules
• README.md — repository overview
• index.md — knowledge base index