# Deployment and Update Process

This page describes how to deploy and update the LLM servers on Dankserver
and Dankclaw. The process uses git as the single source of truth. All script
edits flow through this repository.

> 2026-08-22: the former `vllmScripts` repository is merged into this repo.
> Both LLM services now run from the `/home/dank/serverAppConfigs` checkout
> on each host. The retired checkouts (`/home/dank/vLLM` on Dankserver,
> `/home/dank/vllmScripts` on Dankclaw) are archived on the hosts.

## Overview

Two servers run LLM inference services. Each server pulls its scripts from
this repository. The deployment process moves a committed change from the
repository to a running service.

| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 7770 |

## Workflow

The deployment workflow has seven steps.

```mermaid
graph LR
    A[Edit scripts locally] --> B[Commit]
    B --> C[Push to origin<br>Gitea on Dankclaw]
    C --> D[Pull on target<br>git pull]
    D --> E[Restart podman stack]
    E --> F[Relink + restart LLM unit]
    F --> G[Verify health endpoints]
```

1. Fetch the current script from the target machine.
2. Commit the fetched version in this repository.
3. Edit the script locally in the working tree.
4. Run the tests (`./tests/run.sh` must pass).
5. Commit and push the change to origin.
6. Pull the change on the target machine.
7. Restart the services and verify that they are healthy.

## Deploy Script

The `scripts/deploy.sh` script is the unified deployer. One run per target
covers both workloads on that host: the podman stack and the LLM service.
Run it from the repository root.

Run this command to deploy to Dankserver:

```
./scripts/deploy.sh dankserver
```

Run this command to deploy to Dankclaw:

```
./scripts/deploy.sh dankclaw
```

Add the `--dry-run` flag to preview without making changes:

```
./scripts/deploy.sh dankserver --dry-run
```

The script does these actions:

1. Verifies that the local working tree has no uncommitted changes.
2. Pushes the `main` branch to origin.
3. Pulls the repository on the target machine.
4. Restarts the podman stack (Dankclaw only. The Dankserver media stack
   runs as bare containers).
5. Relinks the LLM systemd units, reloads systemd, enables, and restarts
   the active LLM service.
6. Waits for the Caddy homepage and the LLM health endpoint (over ssh).

Set `FRANK_SUDO_PW` when running headless on a host without passwordless
sudo, so that remote sudo reads the password from stdin (`sudo -S`). The
password is never stored in this repo. Both fleet hosts currently allow
passwordless sudo for the deploy user (verified 2026-08-22). The
passwordless restart helpers from the retired topology are obsolete and are
not used.

Health endpoints:

- Dankserver: `http://127.0.0.1:7770/health`
- Dankclaw: `http://127.0.0.1:7770/health`

## Manual Update

### Update Dankserver (frank-default-small)

Run these commands on Dankserver:

```
cd /home/dank/serverAppConfigs
git pull origin main
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-small
curl -sf http://127.0.0.1:7770/health
```

### Update Dankclaw (frank-default-large)

Run these commands on Dankclaw:

```
cd /home/dank/serverAppConfigs
git pull origin main
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:7770/health
```

## Service Units

### Dankserver — frank-default-small

The active service on Dankserver is upstream llama.cpp (b10121) serving
`Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q6_K_P.gguf` under the
alias `frank-default-small` on port 7770. It runs the stock
`/home/dank/llama.cpp/build/bin/llama-server` binary with the 38 MoE
experts on CPU (`--n-cpu-moe 38`) and 7 CPU threads, ngram-mod
speculative decoding (n_min 48 / n_max 64 / n_match 24), f16 KV for both
K and V, context 786432 (3 slots of ~262K each), batch 4096 / ubatch 2048,
`--no-mmap`. The unit file is `scripts/frank-default-small.service`. It
launches through `scripts/serve-frank-small.sh`, which reads tunables from
`scripts/frank-default-small.env`.

### Dankclaw — frank-default-large

The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v3 five-file payload: GGUF
+ FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj) under the alias
`frank-default-large` on port 7770 (moved from 52625 on 2026-08-22). It runs
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP
(`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context
(1 slot. Concurrent requests queue. The server never preempts a running
generation), batch 4096 / ubatch 4096, 16 context checkpoints, 8 GiB
prompt cache, and reasoning in deepseek format. The unit file is
`scripts/frank-default-large.service`. It launches through
`scripts/serve-frank-large.sh`, which reads tunables from
`scripts/frank-default-large.env`. See [[ciru-activefpx-promptforge]].

> Restored 2026-08-21. The intermediate DFlash2 Vulkan deployment was
> retired the same day. See [[dflash2-large-vulkan]] for the archived runbook.

## Validation

The deploy script checks the health endpoint after a restart. For deeper
checks, run a completion request against the endpoint:

```
curl -s http://127.0.0.1:7770/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"frank-default-small","messages":[{"role":"user","content":"Say OK"}]}'
```

## Troubleshooting

If the health check fails after a deploy:

1. Read the service logs with `journalctl -u <service> -n 200`.
2. Confirm that the model files exist in the cache.
3. Confirm that the GPU has free memory (`nvidia-smi` on Dankserver, `rocm-smi` on Dankclaw).
4. Check the service status with `systemctl status <service>`.

## Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13)

On 2026-08-13 the old `frank-default-large` service crash-looped for hours
on Dankclaw. Root cause: the env files exported `TEMP=1.0` (a sampling
default). The ROCm HIP runtime reads `TEMP` as its JIT kernel-cache
directory, used the path `1.0`, failed, and segfaulted on a NULL object.
Fix: rename the sampling variable to `TEMPERATURE` (commit `227ce01`).

If a service segfaults at load with no obvious cause, compare the exported
env against the HIP runtime's env reads. Do not name any tunable `TEMP`,
`HOST`, or `GPU_MAX_HEAP_SIZE`.

## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)

Historical: on the retired topology, parallel jobs on the Dankserver 35B
service died with a cuBLAS crash under researcher load. The affected
service is retired. A client abort logs `W srv stop: cancel task,
id_task = N`. A natural stop logs only `slot release` with
`truncated = 0`. The server never preempts a running generation.

## Incident: frank-default-small Crash Loop After Distro Upgrade (2026-08-21)

On 2026-08-21 a release upgrade moved Dankserver from Ubuntu 24.04 to
Ubuntu 26.04 (Resolute Raccoon). The upgrade removed the CUDA 12 runtime
packages: `libcudart12`, `libcublas12`, `libcublaslt12`, and
`nvidia-cuda-toolkit`. The `llama-server` binary links
`libcudart.so.12` and `libcublas.so.12`. On every start the binary exited
with code 127. systemd restart-looped the service continuously. The health
endpoint on port 7770 stayed down.

Fix: reinstall the runtime packages. The same versions remain in the 26.04
repository.

    sudo apt-get install -y libcudart12 libcublas12 libcublaslt12

Verify that the binary resolves all libraries:

    ldd /home/dank/llama.cpp/build/bin/llama-server | grep "not found"

Start the service and check the health endpoint on port 7770.

After any release upgrade on Dankserver, run the `ldd` check above. A
missing CUDA 12 library means the packages need a reinstall.

## See Also

- [AGENTS.md](../../AGENTS.md) — the git workflow rules
- [README.md](../../README.md) — repository overview
- [index.md](index.md) — LLM fleet knowledge base index
- [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) — the full redo runbook
- [docs/index.md](../index.md) — the project knowledge base (podman stacks)