serverAppConfigs/docs/deployment.md

# Deployment and Update Process

This page describes how to deploy and update the LLM servers on Dankserver and Dankclaw. The process uses git as the single source of truth. All script edits flow through this repository.

## Overview

Two servers run LLM inference services. Each server pulls its scripts from this repository. The deployment process moves a committed change from the repository to a running service.

| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B-Uncensored Q6_K, split 0.68 to the Dankserver worker) | frank-default-large | 7770 |
| Dankclaw | dank@dankclaw | FastFlowLM NPU (qwythos:9b, 262144 ctx) | frank-default-small | 52625 |
| Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090) | frank-pp-worker | 50052 |

The `frank-default-small` (NPU) unit is not in `deploy.sh` and is disabled for boot (see Boot Policy below). If the unit is still loaded (pre-reboot), restart it by hand with `sudo systemctl restart frank-default-small`. After a reboot, re-enable the unit first (see Boot Policy).

## Boot Policy (2026-08-17)

Only the pipeline-parallel pair starts at boot: `frank-default-large` (Dankclaw head) and `frank-pp-worker` (Dankserver worker). Every other inference unit is disabled. This was set on 2026-08-17.

Disabled units:

| Machine | Unit | Fate |
|---------|------|------|
| Dankclaw | `frank-default-small` (FastFlowLM NPU, port 52625) | Disabled; unit file removed from `/etc/systemd/system`. The service keeps running until stopped or rebooted. |
| Dankclaw | `frank-large-ling` (Ling trial) | Disabled. |
| Dankclaw | `vllm-thinkingcap` | Disabled (already). |
| Dankserver | `frank-default-small` (old llama.cpp 35B) | Disabled; unit file removed. |
| Dankserver | `llamaserver`, `ollama`, `vllm-awq-qwen36`, `vllm-qwen36` | Disabled. |

`systemctl disable` removes the symlinked unit file from `/etc/systemd/system` when the file is a symlink. The repo copies stay intact. To restore a disabled service, re-symlink the unit and enable it:

```
sudo ln -sf /home/dank/<repo>/scripts/<unit>.service /etc/systemd/system/<unit>.service
sudo systemctl daemon-reload
sudo systemctl enable <unit>
```

The NPU re-enable recipe uses the Dankclaw repo path `/home/dank/vllmScripts/scripts/frank-default-small.service`.

The Dankserver helper `frank-restart-small.sh` targets the removed `frank-default-small` unit. Do not use it. Restart the PP worker with an explicit command:

```
sudo systemctl restart frank-pp-worker
```

`scripts/deploy.sh dankserver` also targets the removed unit. Do not run it until the script is retargeted to `frank-pp-worker`. The `dankclaw` target works: it restarts the PP head through `frank-restart-large.sh`.

## Workflow

The deployment workflow has six steps.

```mermaid
graph LR
    A[Edit scripts locally] --> B[Commit]
    B --> C[Push to origin<br>Gitea on Dankclaw]
    C --> D[Pull on target<br>git pull]
    D --> E[Restart systemd service]
    E --> F[Verify health endpoint]
```

1. Fetch the current script from the target machine.
2. Commit the fetched version in this repository.
3. Edit the script locally in the working tree.
4. Commit and push the change to origin.
5. Pull the change on the target machine.
6. Restart the service and verify that it is healthy.

## Deploy Script

The `scripts/deploy.sh` script executes steps 4 through 6. Run it from the repository root.

Run this command to deploy to Dankclaw:

```
./scripts/deploy.sh dankclaw
```

Run this command to deploy to Dankserver:

```
./scripts/deploy.sh dankserver
```

Add the `--dry-run` flag to preview without making changes:

```
./scripts/deploy.sh dankclaw --dry-run
```

The script does these actions:

1. Verifies that the local working tree has no uncommitted changes.
2. Pushes the `main` branch to origin.
3. Pulls the repository on the target machine.
4. Restarts the service. When the passwordless helper is installed (see below), it runs the helper with plain `sudo`. Otherwise it falls back to the inline sudo commands (unit symlink, cache drop, daemon-reload, restart).
5. Waits for the health endpoint to respond (over ssh).

Set `FRANK_SUDO_PW` when running headless on a target that does not yet have the passwordless helper, so that remote sudo reads the password from stdin:

```
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw
```

## Passwordless Restart

Each target has a root-owned helper script that performs the full restart. The user `dank` runs it with `sudo` and no password via a sudoers drop-in.

| Target | Helper | Sudoers drop-in |
|--------|--------|-----------------|
| Dankclaw | `/usr/local/sbin/frank-restart-large.sh` | `/etc/sudoers.d/frank-large` |
| Dankserver | `/usr/local/sbin/frank-restart-small.sh` (obsolete, see below) | `/etc/sudoers.d/frank-small` |

The Dankclaw helper does these steps:

1. Runs `sync`.
2. Drops the page caches with `echo 3 > /proc/sys/vm/drop_caches`.
3. Symlinks the systemd unit.
4. Reloads systemd.
5. Enables and restarts the service.

The Dankserver helper `frank-restart-small.sh` is obsolete: it restarts the removed `frank-default-small` unit. Do not use it. The sudoers drop-in does not cover `frank-pp-worker`, so the worker restart needs the password (or a new helper). Restart the worker with `sudo systemctl restart frank-pp-worker`.

The sudoers drop-in contains exactly two rules:

```
dank ALL=(root) NOPASSWD: /usr/local/sbin/frank-restart-large.sh
dank ALL=(root) NOPASSWD: /usr/bin/systemctl status frank-default-large
```

Rules match the command and its arguments exactly. Do not append extra arguments. For example, `sudo systemctl status frank-default-large --no-pager` fails; use the exact form instead. Install or replace the helper and drop-in only through `sudo` on the target, and validate with `sudo visudo -c`.

To test that the setup works, run this command:

```
sudo -n /usr/local/sbin/frank-restart-large.sh
```

### Update Dankclaw

1. Pull the repository.
2. Symlink the unit file, if it changed.
3. Drop the page caches.
4. Reload and restart the service.

Run these commands on Dankclaw:

```
cd /home/dank/vllmScripts
git pull origin main
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
sudo systemctl daemon-reload
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:7770/health
```

### Update Dankserver

Run these commands on Dankserver to update and restart the PP worker:

```
cd /home/dank/vLLM
git pull origin main
sudo ln -sf /home/dank/vLLM/scripts/frank-pp-worker.service /etc/systemd/system/frank-pp-worker.service
sudo systemctl daemon-reload
sudo systemctl restart frank-pp-worker
```

The worker has no HTTP health endpoint. Verify the listener instead:

```
ss -tln | grep :50052
```

The old passwordless helper `frank-restart-small.sh` targets the removed `frank-default-small` unit. Do not use it. See Boot Policy.

## Service Units

### Dankclaw — frank-default-large

The active service on Dankclaw is the llama.cpp **pipeline-parallel
head**. It serves `Qwen3.8-27B-Uncensored-Q6_K.gguf` (JonathanColetti,
imatrix) under the alias `frank-default-large` on port 7770. The head
splits the tensors `0.68,0.32` to the `ggml-rpc` worker on Dankserver
(RTX 3090, port 50052): 45 of 65 blocks (0-44) on the 3090 — the
maximum for this config (measured 2026-08-17; see the PP doc).
Speculative decoding runs **MTP** with a standalone draft head
(`--spec-draft-model`, `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`, n_max
4) plus the `--spec-default` ngram-mod fallback (48/64/24). The head GPU
side runs **ROCm0** (no `--device` flag; default device selection puts
RPC first and picks ROCm for the iGPU). **1 slot** at 262144 context
(pool 262144, `n_ctx_slot = 262144`), KV cache **K q8_0 + V f16**
(mismatched, 784 MiB per FA layer; RPC0 8624 + ROCm0 3920 MiB) with
Flash Attention retained via the GGML_CUDA_FA_ALL_QUANTS rebuild, full
layer offload (`--gpu-layers all`, `--fit off`), batch 8192 / ubatch
256, `--direct-io`. The unit file is
`scripts/frank-default-large.service`. It launches through
`scripts/serve-pp-head.sh`, which reads tunables from
`scripts/frank-default-large.env`.

> Note: 2026-08-17 — the Q6_K swap: model + standalone draft head,
> mismatched q8_0/f16 KV with FA (both binaries rebuilt with
> GGML_CUDA_FA_ALL_QUANTS at 9b05354ec), split 0.68 (was 0.53 with the
> Q8_K_XL). Worker at 23854/24576 MiB, ~273 MiB free. MTP acceptance
> 0.75-0.83 (mean len 4.0-4.3), stronger than the old bundled head.

> Note: 2026-08-17 — single-slot config: `PARALLEL=1`, pool 262144
> (was 2 slots x 200700 at 401400), split 0.53 (was 0.40). The 3090
> landed at 23460/24576 MiB, ~667 MiB free. Same-day incident: a full
> Dankserver disk corrupted the worker RPC tensor cache -> degenerate
> output; fixed by clearing the cache + freeing the disk. Guard:
> `scripts/prune-rpc-cache.sh` (cron, 40 GiB cap).

> Note: 2026-08-16 — the pipeline-parallel split was promoted to
> production as `frank-default-large` (renamed from `frank-pp-head`);
> the standalone single-node service retired. MTP was re-enabled at
> `n_max 4` over the ngram-mod fallback. See
> [[Pipeline Parallelism/Qwen3.8 PP Deployment]] for the runbook.

> Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k
> prompts on ROCm, `ROCm error: an illegal memory access`, status 6/ABRT) →
> `--spec-default` (ngram-mod, parallel-safe, PARALLEL=2). The backend went
> ROCm → Vulkan0 after a 262k-ctx bench showed Vulkan decodes 48% faster
> (5.68 vs 3.83 t/s). That Vulkan0 verdict applied to the standalone
> configuration; the PP head runs ROCm0 by default (HIP registers before
> Vulkan). See [[qwen3.8-spec-matrix]] for both decisions and the bench
> table. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash
> runtime, and the llama-serve/vLLM units on Dankclaw.

### Dankserver — old llama.cpp 35B service (retired 2026-08-16)

Dankserver previously ran `frank-default-small`: llama.cpp serving the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` (ngram-mod spec, 4 slots x 175104 ctx, 36 of 40 MoE expert layers on CPU, mmproj vision). The service was verified `inactive` on 2026-08-16, and its unit file was removed on 2026-08-17 under the boot policy. The name `frank-default-small` now belongs to the FastFlowLM NPU endpoint on Dankclaw (see [[FastFlowLM NPU Inference]]). The RTX 3090 now runs the PP worker. See git history for the full retired configuration.

## Validation

The deploy script checks the health endpoint after a restart. For deeper checks, run the validation script on the target.

Run this command on Dankserver to test vLLM throughput:

```
./scripts/validate-vllm.sh
```

## Troubleshooting

If the health check fails after a deploy:

1. Read the service logs with `journalctl -u <service> -n 200`.
2. Confirm that the model files exist in the cache.
3. Confirm that the GPU has free memory.
4. Check the service status with `systemctl status <service>`.

## Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13)

On 2026-08-13 the `frank-default-large` service crash-looped for hours. Every restart died with SIGSEGV about 20 seconds into model load, at a fixed offset in `libamdhip64` (NULL refcount unref during `ggml_backend_cuda_buffer_set_tensor`). The kernel log showed bursts of `amdgpu: SVM mapping failed, exceeds resident system memory limit`, which pointed at memory but was a red herring.

Root cause: the env files exported `TEMP=1.0` (a sampling default added in commit `db3eb01`). The ROCm HIP runtime reads the `TEMP` environment variable as its JIT kernel-cache directory. HIP then used the path `1.0`, failed, and segfaulted on a NULL object. The crash was deterministic whenever the serve script sourced the env file, and absent when it did not.

Fix: rename the sampling variable to `TEMPERATURE` in the env files and serve scripts (commit `227ce01`). The serve script still passes `--temp "$TEMPERATURE"` to `llama-server`, so sampling defaults are unchanged.

If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Check with `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|..."`. Do not name any tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`.

## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)

A user reported that a streaming job on `frank-default-large` was released early when another job started. The suspicion was speculative decoding. The investigation found three facts:

1. **The server does not preempt.** `llama-server` defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically with concurrent streaming jobs.
2. **The releases were client-side aborts.** The pi coding agent aborts a stream when the user interrupts it. The client records `stopReason: "aborted"`; the server logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`.
3. **Dankserver crashed under the parallel researcher load.** Five parallel research subagents hammered `frank-default-small` with back-to-back requests. The service aborted twice with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. That crash, not speculative decoding, killed the parallel jobs. The affected service is retired; see [[dankserver-vram-growth]] for the VRAM/RAM growth findings behind the crash class.

## See Also

- [AGENTS.md](../AGENTS.md) — the git workflow rules
- [README.md](../README.md) — repository overview
- [index.md](index.md) — knowledge base index
- [serverAppConfigs knowledge base](../serverAppConfigs/docs/index.md) —
  the sibling Podman app stack on the same hosts; its OpenWebUI consumes
  the `:7770` endpoint, and its Gitea instance hosts this repository