serverAppConfigs/docs/deployment.md

Deployment and Update Process

This page describes how to deploy and update the LLM servers on Dankserver and Dankclaw. The process uses git as the single source of truth. All script edits flow through this repository.

Overview

Two servers run LLM inference services. Each server pulls its scripts from this repository. The deployment process moves a committed change from the repository to a running service.

Target	Host	Runtime	Service	Port
Dankclaw	dank@dankclaw	llama.cpp PP head (Qwen3.8-27B UD-Q8_K_XL)	frank-default-large	7770
Dankclaw	dank@dankclaw	FastFlowLM NPU (qwen3.6-moe:35b-a3b)	frank-default-small	52625
Dankserver	dank@dankserver	ggml-rpc worker (RTX 3090)	frank-pp-worker	50052

The frank-default-small (NPU) unit is not in deploy.sh. Restart it by hand with sudo systemctl restart frank-default-small.

Workflow

The deployment workflow has six steps.

graph LR
    A[Edit scripts locally] --> B[Commit]
    B --> C[Push to origin<br>Gitea on Dankclaw]
    C --> D[Pull on target<br>git pull]
    D --> E[Restart systemd service]
    E --> F[Verify health endpoint]

1. Fetch the current script from the target machine.
2. Commit the fetched version in this repository.
3. Edit the script locally in the working tree.
4. Commit and push the change to origin.
5. Pull the change on the target machine.
6. Restart the service and verify that it is healthy.

Deploy Script

The scripts/deploy.sh script executes steps 4 through 6. Run it from the repository root.

Run this command to deploy to Dankclaw:

./scripts/deploy.sh dankclaw

Run this command to deploy to Dankserver:

./scripts/deploy.sh dankserver

Add the --dry-run flag to preview without making changes:

./scripts/deploy.sh dankclaw --dry-run

The script does these actions:

1. Verifies that the local working tree has no uncommitted changes.
2. Pushes the main branch to origin.
3. Pulls the repository on the target machine.
4. Restarts the service. When the passwordless helper is installed (see below), it runs the helper with plain sudo. Otherwise it falls back to the inline sudo commands (unit symlink, cache drop, daemon-reload, restart).
5. Waits for the health endpoint to respond (over ssh).

Set FRANK_SUDO_PW when running headless on a target that does not yet have the passwordless helper, so that remote sudo reads the password from stdin:

FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw

Passwordless Restart

Each target has a root-owned helper script that performs the full restart. The user dank runs it with sudo and no password via a sudoers drop-in.

Target	Helper	Sudoers drop-in
Dankclaw	/usr/local/sbin/frank-restart-large.sh	/etc/sudoers.d/frank-large
Dankserver	/usr/local/sbin/frank-restart-small.sh	/etc/sudoers.d/frank-small

The Dankclaw helper does these steps:

1. Runs sync.
2. Drops the page caches with echo 3 > /proc/sys/vm/drop_caches.
3. Symlinks the systemd unit.
4. Reloads systemd.
5. Enables and restarts the service.

The Dankserver helper does the same steps without the cache drop.

The sudoers drop-in contains exactly two rules:

dank ALL=(root) NOPASSWD: /usr/local/sbin/frank-restart-large.sh
dank ALL=(root) NOPASSWD: /usr/bin/systemctl status frank-default-large

Rules match the command and its arguments exactly. Do not append extra arguments. For example, sudo systemctl status frank-default-large --no-pager fails; use the exact form instead. Install or replace the helper and drop-in only through sudo on the target, and validate with sudo visudo -c.

To test that the setup works, run this command:

sudo -n /usr/local/sbin/frank-restart-large.sh

Update Dankclaw

1. Pull the repository.
2. Symlink the unit file, if it changed.
3. Drop the page caches.
4. Reload and restart the service.

Run these commands on Dankclaw:

cd /home/dank/vllmScripts
git pull origin main
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
sudo systemctl daemon-reload
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:7770/health

Update Dankserver

Run these commands on Dankserver:

cd /home/dank/vLLM
git pull origin main
sudo ln -sf /home/dank/vLLM/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-small
curl -sf http://127.0.0.1:7770/health

Service Units

Dankclaw — frank-default-large

The active service on Dankclaw runs llama.cpp serving the unsloth Qwen3.8-27B UD-Q8_K_XL model under the alias frank-default-large on port 7770. It uses llama.cpp's --spec-default preset (ngram-mod, no draft model) for speculative decoding on the Vulkan0 backend, with 2 parallel slots at 262144 context each, KV cache q8_0 K / q8_0 V on GPU, full layer offload (--gpu-layers all, --fit off), and batch 8192 / ubatch 256. The unit file is scripts/frank-default-large.service. It launches through scripts/serve-llama-frank-large.sh, which reads tunables from scripts/frank-default-large.env.

Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k prompts on ROCm, ROCm error: an illegal memory access, status 6/ABRT) → --spec-default (ngram-mod, parallel-safe, PARALLEL=2). The backend went ROCm → Vulkan0 after a 262k-ctx bench showed Vulkan decodes 48% faster (5.68 vs 3.83 t/s). See qwen3.8-spec-matrix for both decisions and the bench table. Also retired earlier: bundled MTP, the vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime, and the llama-serve/vLLM units on Dankclaw.

Dankserver — frank-default-small (STOPPED 2026-08-16)

Note (2026-08-16): the frank-default-small alias now belongs to the FastFlowLM NPU endpoint on Dankclaw (port 52625, see FastFlowLM NPU Inference). The llama.cpp Qwen3.6-35B-A3B service on Dankserver described below is stopped (verified inactive 2026-08-16). The text below is kept as history.

The active service on Dankserver runs llama.cpp (upstream, b10108+) and serves the HauhauCS Qwen3.6-35B-A3B uncensored Q6_K_P model under the alias frank-default-small on port 7770. It uses ngram-mod speculative decoding (--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24), 4 parallel slots at 175104 context each (pool 700416), and mmproj vision. 36 of 40 MoE expert layers stay on CPU (--n-cpu-moe 36); 4 expert layers' weights are back on GPU alongside all non-expert tensors (--gpu-layers all). KV stays on GPU (K=f16, V=f16), refitted from 800000 on 2026-08-14 when the pool went dual-f16 (the f16/f16 pool costs 13680 MiB at 700416 cells and did not fit at 800768 cells). Batch 8192 / ubatch 3072, --ctx-checkpoints 4, --no-mmap. The unit file is scripts/frank-default-small.service, symlinked to /etc/systemd/system/. It launches through scripts/serve-llama-frank-small.sh, which reads tunables from scripts/frank-default-small.env.

Note: the Dankserver process currently observed runs --spec-type ngram-mod with --parallel 4, batch 8192 / ubatch 2048, and --n-cpu-moe 34 (this is being bench-tuned by the operator; reconcile against scripts/frank-default-small.env).

Note: this service replaced the retired vllm-awq-qwen36 (AWQ 4-bit, port 34401) and the earlier vllm-qwen36 (FP8) / llama-server (ROCmFPX) units. The repo tracks main on Dankserver at /home/dank/vLLM.

Validation

The deploy script checks the health endpoint after a restart. For deeper checks, run the validation script on the target.

Run this command on Dankserver to test vLLM throughput:

./scripts/validate-vllm.sh

Troubleshooting

If the health check fails after a deploy:

1. Read the service logs with journalctl -u <service> -n 200.
2. Confirm that the model files exist in the cache.
3. Confirm that the GPU has free memory.
4. Check the service status with systemctl status <service>.

Incident: HIP Runtime Crash From the TEMP Env Variable (2026-08-13)

On 2026-08-13 the frank-default-large service crash-looped for hours. Every restart died with SIGSEGV about 20 seconds into model load, at a fixed offset in libamdhip64 (NULL refcount unref during ggml_backend_cuda_buffer_set_tensor). The kernel log showed bursts of amdgpu: SVM mapping failed, exceeds resident system memory limit, which pointed at memory but was a red herring.

Root cause: the env files exported TEMP=1.0 (a sampling default added in commit db3eb01). The ROCm HIP runtime reads the TEMP environment variable as its JIT kernel-cache directory. HIP then used the path 1.0, failed, and segfaulted on a NULL object. The crash was deterministic whenever the serve script sourced the env file, and absent when it did not.

Fix: rename the sampling variable to TEMPERATURE in the env files and serve scripts (commit 227ce01). The serve script still passes --temp "$TEMPERATURE" to llama-server, so sampling defaults are unchanged.

If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Check with strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|...". Do not name any tunable TEMP, HOST, or GPU_MAX_HEAP_SIZE.

Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)

The user reported that on frank-default-large (Dankclaw), a job that is streaming tokens gets released early when another job starts streaming. The suspicion was the speculative decoding configuration ("dspark"). Investigation found no dspark in either deployment: Dankclaw runs draft-dflash, Dankserver runs ngram-mod. Three findings emerged:

1. The server does not preempt. llama-server defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically: two concurrent streaming jobs completed normally with dissimilar prompts, similar prompts, and near-identical prompts (high LCP reuse). parallel=3 on Dankclaw serves up to three concurrent jobs and queues the rest.
2. The releases were client-side aborts. The pi coding agent aborts a stream when the user interrupts it (for example, Escape, or starting a new message in a way that cancels the previous one). The client records stopReason: "aborted" with errorMessage: "Operation aborted", and the server logs srv stop: cancel task, id_task = N. The server journal shows a slot release line with no cancel task line when the generation ends normally (EOS or limit).
3. Dankserver crashed under the parallel researcher load. Five parallel research subagents hammered frank-default-small with back-to-back requests (a new task every few seconds for ~15 minutes). At 19:45:44 and again at 19:55:27 the service aborted with CUDA error: an unsupported value or parameter was passed to the function in cublasSgemm_v2 at ggml-cuda.cu:1541. systemd restarted it, and in-flight streams died with Connection error. or terminated, then 503 Loading model during the ~5 minute reload. This crash, not speculative decoding, killed the parallel jobs. The signature is a CUBLAS invalid-value error in a GEMM under heavy concurrent load; root cause on Dankserver (llama.cpp b10121, driver 595.84, MoE experts CPU-offloaded) was not yet determined.

Diagnosis recipe: to tell a client abort from a server release, read the journal around the release time. A client abort shows W srv stop: cancel task, id_task = N. A natural stop shows only slot release with truncated = 0 and the final timings line.

VRAM and RAM growth under load

This crash class recurs on Dankserver. The 2026-08-14 investigation ruled out ctx-checkpoints and ngram-mod as VRAM growth sources and confirmed RAM growth. See dankserver-vram-growth for the full findings, the monitoring setup, and the mitigation options.

See Also

• AGENTS.md — the git workflow rules
• README.md — repository overview
• index.md — knowledge base index