serverAppConfigs/docs/deployment.md

# Deployment and Update Process

This page describes how to deploy and update the LLM servers on Dankserver and Dankclaw. The process uses git as the single source of truth. All script edits flow through this repository.

## Overview

Two servers run LLM inference services. Each server pulls its scripts from this repository. The deployment process moves a committed change from the repository to a running service.

| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B UD-Q8_K_XL) | frank-default-large | 7770 |
| Dankclaw | dank@dankclaw | FastFlowLM NPU (qwen3.6-moe:35b-a3b) | frank-default-small | 52625 |
| Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090) | frank-pp-worker | 50052 |

The `frank-default-small` (NPU) unit is not in `deploy.sh`. Restart it by hand with `sudo systemctl restart frank-default-small`.

## Workflow

The deployment workflow has six steps.

```mermaid
graph LR
    A[Edit scripts locally] --> B[Commit]
    B --> C[Push to origin<br>Gitea on Dankclaw]
    C --> D[Pull on target<br>git pull]
    D --> E[Restart systemd service]
    E --> F[Verify health endpoint]
```

1. Fetch the current script from the target machine.
2. Commit the fetched version in this repository.
3. Edit the script locally in the working tree.
4. Commit and push the change to origin.
5. Pull the change on the target machine.
6. Restart the service and verify that it is healthy.

## Deploy Script

The `scripts/deploy.sh` script executes steps 4 through 6. Run it from the repository root.

Run this command to deploy to Dankclaw:

```
./scripts/deploy.sh dankclaw
```

Run this command to deploy to Dankserver:

```
./scripts/deploy.sh dankserver
```

Add the `--dry-run` flag to preview without making changes:

```
./scripts/deploy.sh dankclaw --dry-run
```

The script does these actions:

1. Verifies that the local working tree has no uncommitted changes.
2. Pushes the `main` branch to origin.
3. Pulls the repository on the target machine.
4. Restarts the service. When the passwordless helper is installed (see below), it runs the helper with plain `sudo`. Otherwise it falls back to the inline sudo commands (unit symlink, cache drop, daemon-reload, restart).
5. Waits for the health endpoint to respond (over ssh).

Set `FRANK_SUDO_PW` when running headless on a target that does not yet have the passwordless helper, so that remote sudo reads the password from stdin:

```
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw
```

## Passwordless Restart

Each target has a root-owned helper script that performs the full restart. The user `dank` runs it with `sudo` and no password via a sudoers drop-in.

| Target | Helper | Sudoers drop-in |
|--------|--------|-----------------|
| Dankclaw | `/usr/local/sbin/frank-restart-large.sh` | `/etc/sudoers.d/frank-large` |
| Dankserver | `/usr/local/sbin/frank-restart-small.sh` | `/etc/sudoers.d/frank-small` |

The Dankclaw helper does these steps:

1. Runs `sync`.
2. Drops the page caches with `echo 3 > /proc/sys/vm/drop_caches`.
3. Symlinks the systemd unit.
4. Reloads systemd.
5. Enables and restarts the service.

The Dankserver helper does the same steps without the cache drop.

The sudoers drop-in contains exactly two rules:

```
dank ALL=(root) NOPASSWD: /usr/local/sbin/frank-restart-large.sh
dank ALL=(root) NOPASSWD: /usr/bin/systemctl status frank-default-large
```

Rules match the command and its arguments exactly. Do not append extra arguments. For example, `sudo systemctl status frank-default-large --no-pager` fails; use the exact form instead. Install or replace the helper and drop-in only through `sudo` on the target, and validate with `sudo visudo -c`.

To test that the setup works, run this command:

```
sudo -n /usr/local/sbin/frank-restart-large.sh
```

### Update Dankclaw

1. Pull the repository.
2. Symlink the unit file, if it changed.
3. Drop the page caches.
4. Reload and restart the service.

Run these commands on Dankclaw:

```
cd /home/dank/vllmScripts
git pull origin main
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
sudo systemctl daemon-reload
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:7770/health
```

### Update Dankserver

Run these commands on Dankserver:

```
cd /home/dank/vLLM
git pull origin main
sudo ln -sf /home/dank/vLLM/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-small
curl -sf http://127.0.0.1:7770/health
```

## Service Units

### Dankclaw — frank-default-large

The active service on Dankclaw is the llama.cpp **pipeline-parallel
head**. It serves the unsloth Qwen3.8-27B `UD-Q8_K_XL` model under the
alias `frank-default-large` on port 7770. The head splits the tensors
`0.40,0.60` to the `ggml-rpc` worker on Dankserver (RTX 3090, port 50052).
Speculative decoding runs **MTP** (bundled head, `--spec-draft-n-max 4`)
with the `--spec-default` ngram-mod fallback (48/64/24). The head GPU side
runs **ROCm0** (no `--device` flag; default device selection puts RPC
first and picks ROCm for the iGPU). **2 parallel slots** at 200704 context
each (pool 401400), KV cache `f16` K / `f16` V, full layer offload
(`--gpu-layers all`, `--fit off`), batch 8192 / ubatch 256, `--direct-io`.
The unit file is `scripts/frank-default-large.service`. It launches through
`scripts/serve-pp-head.sh`, which reads tunables from
`scripts/frank-default-large.env`.

> Note: 2026-08-16 — the pipeline-parallel split was promoted to
> production as `frank-default-large` (renamed from `frank-pp-head`);
> the standalone single-node service retired. MTP was re-enabled at
> `n_max 4` over the ngram-mod fallback. The pool shrank from 524288 to
> 401400 (2 slots x 200700) to fit the split KV budget. See
> [[Pipeline Parallelism/Qwen3.8 PP Deployment]] for the runbook.

> Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k
> prompts on ROCm, `ROCm error: an illegal memory access`, status 6/ABRT) →
> `--spec-default` (ngram-mod, parallel-safe, PARALLEL=2). The backend went
> ROCm → Vulkan0 after a 262k-ctx bench showed Vulkan decodes 48% faster
> (5.68 vs 3.83 t/s). That Vulkan0 verdict applied to the standalone
> configuration; the PP head runs ROCm0 by default (HIP registers before
> Vulkan). See [[qwen3.8-spec-matrix]] for both decisions and the bench
> table. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash
> runtime, and the llama-serve/vLLM units on Dankclaw.

### Dankserver — frank-default-small (STOPPED 2026-08-16)

> Note (2026-08-16): the `frank-default-small` alias now belongs to the FastFlowLM NPU endpoint on Dankclaw (port 52625, see [[FastFlowLM NPU Inference]]). The llama.cpp Qwen3.6-35B-A3B service on Dankserver described below is stopped (verified `inactive` 2026-08-16). The text below is kept as history.

The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 175104 context each (pool 700416), and mmproj vision. **36 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 36`); 4 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`f16`, V=`f16`), refitted from 800000 on 2026-08-14 when the pool went dual-f16 (the f16/f16 pool costs 13680 MiB at 700416 cells and did not fit at 800768 cells). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.

> Note: the Dankserver process currently observed runs `--spec-type ngram-mod` with `--parallel 4`, batch 8192 / ubatch 2048, and `--n-cpu-moe 34` (this is being bench-tuned by the operator; reconcile against `scripts/frank-default-small.env`).

> Note: this service replaced the retired `vllm-awq-qwen36` (AWQ 4-bit, port 34401) and the earlier `vllm-qwen36` (FP8) / `llama-server` (ROCmFPX) units. The repo tracks `main` on Dankserver at `/home/dank/vLLM`.

## Validation

The deploy script checks the health endpoint after a restart. For deeper checks, run the validation script on the target.

Run this command on Dankserver to test vLLM throughput:

```
./scripts/validate-vllm.sh
```

## Troubleshooting

If the health check fails after a deploy:

1. Read the service logs with `journalctl -u <service> -n 200`.
2. Confirm that the model files exist in the cache.
3. Confirm that the GPU has free memory.
4. Check the service status with `systemctl status <service>`.

## Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13)

On 2026-08-13 the `frank-default-large` service crash-looped for hours. Every restart died with SIGSEGV about 20 seconds into model load, at a fixed offset in `libamdhip64` (NULL refcount unref during `ggml_backend_cuda_buffer_set_tensor`). The kernel log showed bursts of `amdgpu: SVM mapping failed, exceeds resident system memory limit`, which pointed at memory but was a red herring.

Root cause: the env files exported `TEMP=1.0` (a sampling default added in commit `db3eb01`). The ROCm HIP runtime reads the `TEMP` environment variable as its JIT kernel-cache directory. HIP then used the path `1.0`, failed, and segfaulted on a NULL object. The crash was deterministic whenever the serve script sourced the env file, and absent when it did not.

Fix: rename the sampling variable to `TEMPERATURE` in the env files and serve scripts (commit `227ce01`). The serve script still passes `--temp "$TEMPERATURE"` to `llama-server`, so sampling defaults are unchanged.

If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Check with `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|..."`. Do not name any tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`.

## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)

The user reported that on `frank-default-large` (Dankclaw), a job that is streaming tokens gets released early when another job starts streaming. The suspicion was the speculative decoding configuration ("dspark"). Investigation found no dspark in either deployment: Dankclaw runs `draft-dflash`, Dankserver runs `ngram-mod`. Three findings emerged:

1. **The server does not preempt.** `llama-server` defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically: two concurrent streaming jobs completed normally with dissimilar prompts, similar prompts, and near-identical prompts (high LCP reuse). `parallel=3` on Dankclaw serves up to three concurrent jobs and queues the rest.
2. **The releases were client-side aborts.** The pi coding agent aborts a stream when the user interrupts it (for example, Escape, or starting a new message in a way that cancels the previous one). The client records `stopReason: "aborted"` with `errorMessage: "Operation aborted"`, and the server logs `srv stop: cancel task, id_task = N`. The server journal shows a `slot release` line with no `cancel task` line when the generation ends normally (EOS or limit).
3. **Dankserver crashed under the parallel researcher load.** Five parallel research subagents hammered `frank-default-small` with back-to-back requests (a new task every few seconds for ~15 minutes). At 19:45:44 and again at 19:55:27 the service aborted with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`. systemd restarted it, and in-flight streams died with `Connection error.` or `terminated`, then `503 Loading model` during the ~5 minute reload. This crash, not speculative decoding, killed the parallel jobs. The signature is a CUBLAS invalid-value error in a GEMM under heavy concurrent load; root cause on Dankserver (llama.cpp b10121, driver 595.84, MoE experts CPU-offloaded) was not yet determined.

Diagnosis recipe: to tell a client abort from a server release, read the journal around the release time. A client abort shows `W srv stop: cancel task, id_task = N`. A natural stop shows only `slot release` with `truncated = 0` and the final timings line.

### VRAM and RAM growth under load

This crash class recurs on Dankserver. The 2026-08-14 investigation ruled out ctx-checkpoints and ngram-mod as VRAM growth sources and confirmed RAM growth. See [[dankserver-vram-growth]] for the full findings, the monitoring setup, and the mitigation options.

## See Also

- [AGENTS.md](../AGENTS.md) — the git workflow rules
- [README.md](../README.md) — repository overview
- [index.md](index.md) — knowledge base index
- [serverAppConfigs knowledge base](../serverAppConfigs/docs/index.md) —
  the sibling Podman app stack on the same hosts; its OpenWebUI consumes
  the `:7770` endpoint, and its Gitea instance hosts this repository