serverAppConfigs/docs/deployment.md

Deployment and Update Process

This page describes how to deploy and update the LLM servers on Dankserver and Dankclaw. The process uses git as the single source of truth. All script edits flow through this repository.

Overview

Two servers run LLM inference services. Each server pulls its scripts from this repository. The deployment process moves a committed change from the repository to a running service.

Target	Host	Runtime	Service	Port
Dankclaw	dank@dankclaw	llama.cpp PP head (Qwen3.8-27B-Uncensored Q6_K, split 0.68 to the Dankserver worker)	frank-default-large	7770
Dankclaw	dank@dankclaw	FastFlowLM NPU (qwythos:9b, 262144 ctx)	frank-default-small	52625
Dankserver	dank@dankserver	ggml-rpc worker (RTX 3090)	frank-pp-worker	50052

The frank-default-small (NPU) unit is not in deploy.sh and is disabled for boot (see Boot Policy below). If the unit is still loaded (pre-reboot), restart it by hand with sudo systemctl restart frank-default-small. After a reboot, re-enable the unit first (see Boot Policy).

Boot Policy (2026-08-17)

Only the pipeline-parallel pair starts at boot: frank-default-large (Dankclaw head) and frank-pp-worker (Dankserver worker). Every other inference unit is disabled. This was set on 2026-08-17.

Disabled units:

Machine	Unit	Fate
Dankclaw	frank-default-small (FastFlowLM NPU, port 52625)	Disabled; unit file removed from /etc/systemd/system. The service keeps running until stopped or rebooted.
Dankclaw	frank-large-ling (Ling trial)	Disabled.
Dankclaw	vllm-thinkingcap	Disabled (already).
Dankserver	frank-default-small (old llama.cpp 35B)	Disabled; unit file removed.
Dankserver	llamaserver, ollama, vllm-awq-qwen36, vllm-qwen36	Disabled.

systemctl disable removes the symlinked unit file from /etc/systemd/system when the file is a symlink. The repo copies stay intact. To restore a disabled service, re-symlink the unit and enable it:

sudo ln -sf /home/dank/<repo>/scripts/<unit>.service /etc/systemd/system/<unit>.service
sudo systemctl daemon-reload
sudo systemctl enable <unit>

The NPU re-enable recipe uses the Dankclaw repo path /home/dank/vllmScripts/scripts/frank-default-small.service.

The Dankserver helper frank-restart-small.sh targets the removed frank-default-small unit. Do not use it. Restart the PP worker with an explicit command:

sudo systemctl restart frank-pp-worker

scripts/deploy.sh dankserver also targets the removed unit. Do not run it until the script is retargeted to frank-pp-worker. The dankclaw target works: it restarts the PP head through frank-restart-large.sh.

Workflow

The deployment workflow has six steps.

graph LR
    A[Edit scripts locally] --> B[Commit]
    B --> C[Push to origin<br>Gitea on Dankclaw]
    C --> D[Pull on target<br>git pull]
    D --> E[Restart systemd service]
    E --> F[Verify health endpoint]

1. Fetch the current script from the target machine.
2. Commit the fetched version in this repository.
3. Edit the script locally in the working tree.
4. Commit and push the change to origin.
5. Pull the change on the target machine.
6. Restart the service and verify that it is healthy.

Deploy Script

The scripts/deploy.sh script executes steps 4 through 6. Run it from the repository root.

Run this command to deploy to Dankclaw:

./scripts/deploy.sh dankclaw

Run this command to deploy to Dankserver:

./scripts/deploy.sh dankserver

Add the --dry-run flag to preview without making changes:

./scripts/deploy.sh dankclaw --dry-run

The script does these actions:

1. Verifies that the local working tree has no uncommitted changes.
2. Pushes the main branch to origin.
3. Pulls the repository on the target machine.
4. Restarts the service. When the passwordless helper is installed (see below), it runs the helper with plain sudo. Otherwise it falls back to the inline sudo commands (unit symlink, cache drop, daemon-reload, restart).
5. Waits for the health endpoint to respond (over ssh).

Set FRANK_SUDO_PW when running headless on a target that does not yet have the passwordless helper, so that remote sudo reads the password from stdin:

FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw

Passwordless Restart

Each target has a root-owned helper script that performs the full restart. The user dank runs it with sudo and no password via a sudoers drop-in.

Target	Helper	Sudoers drop-in
Dankclaw	/usr/local/sbin/frank-restart-large.sh	/etc/sudoers.d/frank-large
Dankserver	/usr/local/sbin/frank-restart-small.sh (obsolete, see below)	/etc/sudoers.d/frank-small

The Dankclaw helper does these steps:

1. Runs sync.
2. Drops the page caches with echo 3 > /proc/sys/vm/drop_caches.
3. Symlinks the systemd unit.
4. Reloads systemd.
5. Enables and restarts the service.

The Dankserver helper frank-restart-small.sh is obsolete: it restarts the removed frank-default-small unit. Do not use it. The sudoers drop-in does not cover frank-pp-worker, so the worker restart needs the password (or a new helper). Restart the worker with sudo systemctl restart frank-pp-worker.

The sudoers drop-in contains exactly two rules:

dank ALL=(root) NOPASSWD: /usr/local/sbin/frank-restart-large.sh
dank ALL=(root) NOPASSWD: /usr/bin/systemctl status frank-default-large

Rules match the command and its arguments exactly. Do not append extra arguments. For example, sudo systemctl status frank-default-large --no-pager fails; use the exact form instead. Install or replace the helper and drop-in only through sudo on the target, and validate with sudo visudo -c.

To test that the setup works, run this command:

sudo -n /usr/local/sbin/frank-restart-large.sh

Update Dankclaw

1. Pull the repository.
2. Symlink the unit file, if it changed.
3. Drop the page caches.
4. Reload and restart the service.

Run these commands on Dankclaw:

cd /home/dank/vllmScripts
git pull origin main
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
sudo systemctl daemon-reload
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:7770/health

Update Dankserver

Run these commands on Dankserver to update and restart the PP worker:

cd /home/dank/vLLM
git pull origin main
sudo ln -sf /home/dank/vLLM/scripts/frank-pp-worker.service /etc/systemd/system/frank-pp-worker.service
sudo systemctl daemon-reload
sudo systemctl restart frank-pp-worker

The worker has no HTTP health endpoint. Verify the listener instead:

ss -tln | grep :50052

The old passwordless helper frank-restart-small.sh targets the removed frank-default-small unit. Do not use it. See Boot Policy.

Service Units

Dankclaw — frank-default-large

The active service on Dankclaw is the llama.cpp pipeline-parallel head. It serves Qwen3.8-27B-Uncensored-Q6_K.gguf (JonathanColetti, imatrix) under the alias frank-default-large on port 7770. The head splits the tensors 0.68,0.32 to the ggml-rpc worker on Dankserver (RTX 3090, port 50052): 45 of 65 blocks (0-44) on the 3090 — the maximum for this config (measured 2026-08-17; see the PP doc). Speculative decoding runs MTP with a standalone draft head (--spec-draft-model, Qwen3.8-27B-Uncensored-draft-Q8_0.gguf, n_max 4) plus the --spec-default ngram-mod fallback (48/64/24). The head GPU side runs ROCm0 (no --device flag; default device selection puts RPC first and picks ROCm for the iGPU). 1 slot at 262144 context (pool 262144, n_ctx_slot = 262144), KV cache K q8_0 + V f16 (mismatched, 784 MiB per FA layer; RPC0 8624 + ROCm0 3920 MiB) with Flash Attention retained via the GGML_CUDA_FA_ALL_QUANTS rebuild, full layer offload (--gpu-layers all, --fit off), batch 8192 / ubatch 256, --direct-io. The unit file is scripts/frank-default-large.service. It launches through scripts/serve-pp-head.sh, which reads tunables from scripts/frank-default-large.env.

Note: 2026-08-17 — the Q6_K swap: model + standalone draft head, mismatched q8_0/f16 KV with FA (both binaries rebuilt with GGML_CUDA_FA_ALL_QUANTS at 9b05354ec), split 0.68 (was 0.53 with the Q8_K_XL). Worker at 23854/24576 MiB, ~273 MiB free. MTP acceptance 0.75-0.83 (mean len 4.0-4.3), stronger than the old bundled head.

Note: 2026-08-17 — single-slot config: PARALLEL=1, pool 262144 (was 2 slots x 200700 at 401400), split 0.53 (was 0.40). The 3090 landed at 23460/24576 MiB, ~667 MiB free. Same-day incident: a full Dankserver disk corrupted the worker RPC tensor cache -> degenerate output; fixed by clearing the cache + freeing the disk. Guard: scripts/prune-rpc-cache.sh (cron, 40 GiB cap).

Note: 2026-08-16 — the pipeline-parallel split was promoted to production as frank-default-large (renamed from frank-pp-head); the standalone single-node service retired. MTP was re-enabled at n_max 4 over the ngram-mod fallback. See Pipeline Parallelism/Qwen3.8 PP Deployment for the runbook.

Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k prompts on ROCm, ROCm error: an illegal memory access, status 6/ABRT) → --spec-default (ngram-mod, parallel-safe, PARALLEL=2). The backend went ROCm → Vulkan0 after a 262k-ctx bench showed Vulkan decodes 48% faster (5.68 vs 3.83 t/s). That Vulkan0 verdict applied to the standalone configuration; the PP head runs ROCm0 by default (HIP registers before Vulkan). See qwen3.8-spec-matrix for both decisions and the bench table. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime, and the llama-serve/vLLM units on Dankclaw.

Dankserver — old llama.cpp 35B service (retired 2026-08-16)

Dankserver previously ran frank-default-small: llama.cpp serving the HauhauCS Qwen3.6-35B-A3B uncensored Q6_K_P (ngram-mod spec, 4 slots x 175104 ctx, 36 of 40 MoE expert layers on CPU, mmproj vision). The service was verified inactive on 2026-08-16, and its unit file was removed on 2026-08-17 under the boot policy. The name frank-default-small now belongs to the FastFlowLM NPU endpoint on Dankclaw (see FastFlowLM NPU Inference). The RTX 3090 now runs the PP worker. See git history for the full retired configuration.

Validation

The deploy script checks the health endpoint after a restart. For deeper checks, run the validation script on the target.

Run this command on Dankserver to test vLLM throughput:

./scripts/validate-vllm.sh

Troubleshooting

If the health check fails after a deploy:

1. Read the service logs with journalctl -u <service> -n 200.
2. Confirm that the model files exist in the cache.
3. Confirm that the GPU has free memory.
4. Check the service status with systemctl status <service>.

Incident: HIP Runtime Crash From the TEMP Env Variable (2026-08-13)

On 2026-08-13 the frank-default-large service crash-looped for hours. Every restart died with SIGSEGV about 20 seconds into model load, at a fixed offset in libamdhip64 (NULL refcount unref during ggml_backend_cuda_buffer_set_tensor). The kernel log showed bursts of amdgpu: SVM mapping failed, exceeds resident system memory limit, which pointed at memory but was a red herring.

Root cause: the env files exported TEMP=1.0 (a sampling default added in commit db3eb01). The ROCm HIP runtime reads the TEMP environment variable as its JIT kernel-cache directory. HIP then used the path 1.0, failed, and segfaulted on a NULL object. The crash was deterministic whenever the serve script sourced the env file, and absent when it did not.

Fix: rename the sampling variable to TEMPERATURE in the env files and serve scripts (commit 227ce01). The serve script still passes --temp "$TEMPERATURE" to llama-server, so sampling defaults are unchanged.

If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Check with strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|...". Do not name any tunable TEMP, HOST, or GPU_MAX_HEAP_SIZE.

Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)

A user reported that a streaming job on frank-default-large was released early when another job started. The suspicion was speculative decoding. The investigation found three facts:

1. The server does not preempt. llama-server defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically with concurrent streaming jobs.
2. The releases were client-side aborts. The pi coding agent aborts a stream when the user interrupts it. The client records stopReason: "aborted"; the server logs W srv stop: cancel task, id_task = N. A natural stop (EOS or limit) logs only slot release with truncated = 0.
3. Dankserver crashed under the parallel researcher load. Five parallel research subagents hammered frank-default-small with back-to-back requests. The service aborted twice with CUDA error: an unsupported value or parameter was passed to the function in cublasSgemm_v2 at ggml-cuda.cu:1541, then status=6/ABRT, then 503 Loading model for ~5 minutes. That crash, not speculative decoding, killed the parallel jobs. The affected service is retired; see dankserver-vram-growth for the VRAM/RAM growth findings behind the crash class.

See Also

• AGENTS.md — the git workflow rules
• README.md — repository overview
• index.md — knowledge base index
• serverAppConfigs knowledge base — the sibling Podman app stack on the same hosts; its OpenWebUI consumes the :7770 endpoint, and its Gitea instance hosts this repository