228 lines
8.9 KiB
Markdown
228 lines
8.9 KiB
Markdown
# Deployment and Update Process
|
|
|
|
This page describes how to deploy and update the LLM servers on Dankserver
|
|
and Dankclaw. The process uses git as the single source of truth. All script
|
|
edits flow through this repository.
|
|
|
|
> 2026-08-22: the former `vllmScripts` repository is merged into this repo.
|
|
> Both LLM services now run from the `/home/dank/serverAppConfigs` checkout
|
|
> on each host. The retired checkouts (`/home/dank/vLLM` on Dankserver,
|
|
> `/home/dank/vllmScripts` on Dankclaw) are archived on the hosts.
|
|
|
|
## Overview
|
|
|
|
Two servers run LLM inference services. Each server pulls its scripts from
|
|
this repository. The deployment process moves a committed change from the
|
|
repository to a running service.
|
|
|
|
| Target | Host | Runtime | Service | Port |
|
|
|--------|------|---------|---------|------|
|
|
| Dankserver | dank@dankserver | upstream llama.cpp (Ornith-1.5-35B-A3B-CRACK Q6_K, native MTP draft, selected MoE experts offloaded via --override-tensor) | frank-default-small | 7770 |
|
|
| Dankclaw | dank@dankclaw | charlie12345/ROCmFPX llama.cpp (kingjones Qwen3.8 Uncensored Q6_0 ROCmFPX AGENT, standalone MTP, 262K ctx / 1 slot) | frank-default-large | 7770 |
|
|
|
|
## Workflow
|
|
|
|
The deployment workflow has seven steps.
|
|
|
|
```mermaid
|
|
graph LR
|
|
A[Edit scripts locally] --> B[Commit]
|
|
B --> C[Push to origin<br>Gitea on Dankclaw]
|
|
C --> D[Pull on target<br>git pull]
|
|
D --> E[Restart podman stack]
|
|
E --> F[Relink + restart LLM unit]
|
|
F --> G[Verify health endpoints]
|
|
```
|
|
|
|
1. Fetch the current script from the target machine.
|
|
2. Commit the fetched version in this repository.
|
|
3. Edit the script locally in the working tree.
|
|
4. Run the tests (`./tests/run.sh` must pass).
|
|
5. Commit and push the change to origin.
|
|
6. Pull the change on the target machine.
|
|
7. Restart the services and verify that they are healthy.
|
|
|
|
## Deploy Script
|
|
|
|
The `scripts/deploy.sh` script is the unified deployer. One run per target
|
|
covers both workloads on that host: the podman stack and the LLM service.
|
|
Run it from the repository root.
|
|
|
|
Run this command to deploy to Dankserver:
|
|
|
|
```
|
|
./scripts/deploy.sh dankserver
|
|
```
|
|
|
|
Run this command to deploy to Dankclaw:
|
|
|
|
```
|
|
./scripts/deploy.sh dankclaw
|
|
```
|
|
|
|
Add the `--dry-run` flag to preview without making changes:
|
|
|
|
```
|
|
./scripts/deploy.sh dankserver --dry-run
|
|
```
|
|
|
|
The script does these actions:
|
|
|
|
1. Verifies that the local working tree has no uncommitted changes.
|
|
2. Pushes the `main` branch to origin.
|
|
3. Pulls the repository on the target machine.
|
|
4. Restarts the podman stack (Dankclaw only. The Dankserver media stack
|
|
runs as bare containers).
|
|
5. Relinks the LLM systemd units, reloads systemd, enables, and restarts
|
|
the active LLM service.
|
|
6. Waits for the Caddy homepage and the LLM health endpoint (over ssh).
|
|
|
|
Set `FRANK_SUDO_PW` when running headless on a host without passwordless
|
|
sudo, so that remote sudo reads the password from stdin (`sudo -S`). The
|
|
password is never stored in this repo. Both fleet hosts currently allow
|
|
passwordless sudo for the deploy user (verified 2026-08-22). The
|
|
passwordless restart helpers from the retired topology are obsolete and are
|
|
not used.
|
|
|
|
Health endpoints:
|
|
|
|
- Dankserver: `http://127.0.0.1:7770/health`
|
|
- Dankclaw: `http://127.0.0.1:7770/health`
|
|
|
|
## Manual Update
|
|
|
|
### Update Dankserver (frank-default-small)
|
|
|
|
Run these commands on Dankserver:
|
|
|
|
```
|
|
cd /home/dank/serverAppConfigs
|
|
git pull origin main
|
|
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart frank-default-small
|
|
curl -sf http://127.0.0.1:7770/health
|
|
```
|
|
|
|
### Update Dankclaw (frank-default-large)
|
|
|
|
Run these commands on Dankclaw:
|
|
|
|
```
|
|
cd /home/dank/serverAppConfigs
|
|
git pull origin main
|
|
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart frank-default-large
|
|
curl -sf http://127.0.0.1:7770/health
|
|
```
|
|
|
|
## Service Units
|
|
|
|
### Dankserver — frank-default-small
|
|
|
|
The active service on Dankserver is upstream llama.cpp (b10121) serving
|
|
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small`
|
|
on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server`
|
|
binary with expert tensors selected by an explicit `--override-tensor`
|
|
regex. Expert tensors from every full-attention block and blocks 35 through 40
|
|
stay on CUDA; other expert tensors are CPU-resident. The service uses 7 CPU
|
|
threads, ngram-mod speculative decoding only, f16 KV, context 491520 (3 slots
|
|
of 163840 each), batch
|
|
4096, ubatch 1024, and `--no-mmap`. The unit is
|
|
`scripts/frank-default-small.service`. It launches through
|
|
`scripts/serve-frank-small.sh`, which reads tunables from
|
|
`scripts/frank-default-small.env`. It uses `ubatch=1024` and the chat template
|
|
from `peculiar-ragdoll/Qwen-Sharp-Chat-Templates`. See [[dealign-ornith-crack]].
|
|
|
|
### Dankclaw — frank-default-large
|
|
|
|
The active service on Dankclaw uses the ROCmFPX llama.cpp runtime to serve
|
|
[kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF](https://huggingface.co/kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF)
|
|
under the alias `frank-default-large` on port 7770. The main model is
|
|
`Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf`; the standalone MTP draft is
|
|
`mtp-Qwen3.8-27B-Q4_0.gguf`, and the Q8_0 vision projector is loaded alongside
|
|
it. The K-side KV cache remains `f16`; the V-side cache uses `turbo4`. The
|
|
service uses the 262144 context / 1-slot profile. The unit file is
|
|
`scripts/frank-default-large.service`; it launches through
|
|
`scripts/serve-frank-large.sh`, which reads tunables from
|
|
`scripts/frank-default-large.env`, applying the Qwen-Sharp chat template.
|
|
|
|
> The Q8_0 AGENT experiment was retired on 2026-09-01 after repeated
|
|
> `set_rows` crashes and device-local allocation failures. The former caf-eemin
|
|
> ROCmI4 embedded-MTP model and the v3 PromptForge payload are historical
|
|
> deployments. The intermediate DFlash2 Vulkan deployment was retired on
|
|
> 2026-08-21. See [[kingjones-q8agent-rocmfpx]] and [[dflash2-large-vulkan]].
|
|
|
|
## Validation
|
|
|
|
The deploy script checks the health endpoint after a restart. For deeper
|
|
checks, run a completion request against the endpoint:
|
|
|
|
```
|
|
curl -s http://127.0.0.1:7770/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model":"frank-default-small","messages":[{"role":"user","content":"Say OK"}]}'
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
If the health check fails after a deploy:
|
|
|
|
1. Read the service logs with `journalctl -u <service> -n 200`.
|
|
2. Confirm that the model files exist in the cache.
|
|
3. Confirm that the GPU has free memory (`nvidia-smi` on Dankserver, `rocm-smi` on Dankclaw).
|
|
4. Check the service status with `systemctl status <service>`.
|
|
|
|
## Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13)
|
|
|
|
On 2026-08-13 the old `frank-default-large` service crash-looped for hours
|
|
on Dankclaw. Root cause: the env files exported `TEMP=1.0` (a sampling
|
|
default). The ROCm HIP runtime reads `TEMP` as its JIT kernel-cache
|
|
directory, used the path `1.0`, failed, and segfaulted on a NULL object.
|
|
Fix: rename the sampling variable to `TEMPERATURE` (commit `227ce01`).
|
|
|
|
If a service segfaults at load with no obvious cause, compare the exported
|
|
env against the HIP runtime's env reads. Do not name any tunable `TEMP`,
|
|
`HOST`, or `GPU_MAX_HEAP_SIZE`.
|
|
|
|
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
|
|
|
|
Historical: on the retired topology, parallel jobs on the Dankserver 35B
|
|
service died with a cuBLAS crash under researcher load. The affected
|
|
service is retired. A client abort logs `W srv stop: cancel task,
|
|
id_task = N`. A natural stop logs only `slot release` with
|
|
`truncated = 0`. The server never preempts a running generation.
|
|
|
|
## Incident: frank-default-small Crash Loop After Distro Upgrade (2026-08-21)
|
|
|
|
On 2026-08-21 a release upgrade moved Dankserver from Ubuntu 24.04 to
|
|
Ubuntu 26.04 (Resolute Raccoon). The upgrade removed the CUDA 12 runtime
|
|
packages: `libcudart12`, `libcublas12`, `libcublaslt12`, and
|
|
`nvidia-cuda-toolkit`. The `llama-server` binary links
|
|
`libcudart.so.12` and `libcublas.so.12`. On every start the binary exited
|
|
with code 127. systemd restart-looped the service continuously. The health
|
|
endpoint on port 7770 stayed down.
|
|
|
|
Fix: reinstall the runtime packages. The same versions remain in the 26.04
|
|
repository.
|
|
|
|
sudo apt-get install -y libcudart12 libcublas12 libcublaslt12
|
|
|
|
Verify that the binary resolves all libraries:
|
|
|
|
ldd /home/dank/llama.cpp/build/bin/llama-server | grep "not found"
|
|
|
|
Start the service and check the health endpoint on port 7770.
|
|
|
|
After any release upgrade on Dankserver, run the `ldd` check above. A
|
|
missing CUDA 12 library means the packages need a reinstall.
|
|
|
|
## See Also
|
|
|
|
- [AGENTS.md](../../AGENTS.md) — the git workflow rules
|
|
- [README.md](../../README.md) — repository overview
|
|
- [index.md](index.md) — LLM fleet knowledge base index
|
|
- [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) — the full redo runbook
|
|
- [docs/index.md](../index.md) — the project knowledge base (podman stacks)
|