Frank
7ba1acdbc5
3 slots x 229376 (262144 - 32768) = 688128 total context. Updates the .env, models.json (contextWindow/maxTokens 229376 per slot), and the index/deployment/ornith runbook references.
229 lines
8.8 KiB
Markdown
229 lines
8.8 KiB
Markdown
# Deployment and Update Process
|
|
|
|
This page describes how to deploy and update the LLM servers on Dankserver
|
|
and Dankclaw. The process uses git as the single source of truth. All script
|
|
edits flow through this repository.
|
|
|
|
> 2026-08-22: the former `vllmScripts` repository is merged into this repo.
|
|
> Both LLM services now run from the `/home/dank/serverAppConfigs` checkout
|
|
> on each host. The retired checkouts (`/home/dank/vLLM` on Dankserver,
|
|
> `/home/dank/vllmScripts` on Dankclaw) are archived on the hosts.
|
|
|
|
## Overview
|
|
|
|
Two servers run LLM inference services. Each server pulls its scripts from
|
|
this repository. The deployment process moves a committed change from the
|
|
repository to a running service.
|
|
|
|
| Target | Host | Runtime | Service | Port |
|
|
|--------|------|---------|---------|------|
|
|
| Dankserver | dank@dankserver | upstream llama.cpp (Ornith-1.5-35B-A3B-CRACK Q8_0, native MTP draft, MoE experts offloaded via --override-tensor) | frank-default-small | 7770 |
|
|
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (Qwen3.8-27B-Uncensored-ROCmFP4 Q6 AGENT, standalone MTP Q4_0 head, 262K ctx / 1 slot) | frank-default-large | 7770 |
|
|
|
|
## Workflow
|
|
|
|
The deployment workflow has seven steps.
|
|
|
|
```mermaid
|
|
graph LR
|
|
A[Edit scripts locally] --> B[Commit]
|
|
B --> C[Push to origin<br>Gitea on Dankclaw]
|
|
C --> D[Pull on target<br>git pull]
|
|
D --> E[Restart podman stack]
|
|
E --> F[Relink + restart LLM unit]
|
|
F --> G[Verify health endpoints]
|
|
```
|
|
|
|
1. Fetch the current script from the target machine.
|
|
2. Commit the fetched version in this repository.
|
|
3. Edit the script locally in the working tree.
|
|
4. Run the tests (`./tests/run.sh` must pass).
|
|
5. Commit and push the change to origin.
|
|
6. Pull the change on the target machine.
|
|
7. Restart the services and verify that they are healthy.
|
|
|
|
## Deploy Script
|
|
|
|
The `scripts/deploy.sh` script is the unified deployer. One run per target
|
|
covers both workloads on that host: the podman stack and the LLM service.
|
|
Run it from the repository root.
|
|
|
|
Run this command to deploy to Dankserver:
|
|
|
|
```
|
|
./scripts/deploy.sh dankserver
|
|
```
|
|
|
|
Run this command to deploy to Dankclaw:
|
|
|
|
```
|
|
./scripts/deploy.sh dankclaw
|
|
```
|
|
|
|
Add the `--dry-run` flag to preview without making changes:
|
|
|
|
```
|
|
./scripts/deploy.sh dankserver --dry-run
|
|
```
|
|
|
|
The script does these actions:
|
|
|
|
1. Verifies that the local working tree has no uncommitted changes.
|
|
2. Pushes the `main` branch to origin.
|
|
3. Pulls the repository on the target machine.
|
|
4. Restarts the podman stack (Dankclaw only. The Dankserver media stack
|
|
runs as bare containers).
|
|
5. Relinks the LLM systemd units, reloads systemd, enables, and restarts
|
|
the active LLM service.
|
|
6. Waits for the Caddy homepage and the LLM health endpoint (over ssh).
|
|
|
|
Set `FRANK_SUDO_PW` when running headless on a host without passwordless
|
|
sudo, so that remote sudo reads the password from stdin (`sudo -S`). The
|
|
password is never stored in this repo. Both fleet hosts currently allow
|
|
passwordless sudo for the deploy user (verified 2026-08-22). The
|
|
passwordless restart helpers from the retired topology are obsolete and are
|
|
not used.
|
|
|
|
Health endpoints:
|
|
|
|
- Dankserver: `http://127.0.0.1:7770/health`
|
|
- Dankclaw: `http://127.0.0.1:7770/health`
|
|
|
|
## Manual Update
|
|
|
|
### Update Dankserver (frank-default-small)
|
|
|
|
Run these commands on Dankserver:
|
|
|
|
```
|
|
cd /home/dank/serverAppConfigs
|
|
git pull origin main
|
|
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart frank-default-small
|
|
curl -sf http://127.0.0.1:7770/health
|
|
```
|
|
|
|
### Update Dankclaw (frank-default-large)
|
|
|
|
Run these commands on Dankclaw:
|
|
|
|
```
|
|
cd /home/dank/serverAppConfigs
|
|
git pull origin main
|
|
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart frank-default-large
|
|
curl -sf http://127.0.0.1:7770/health
|
|
```
|
|
|
|
## Service Units
|
|
|
|
### Dankserver — frank-default-small
|
|
|
|
The active service on Dankserver is upstream llama.cpp (b10121) serving
|
|
`Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf` under the alias `frank-default-small`
|
|
on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server`
|
|
binary with the MoE expert weights pinned to CPU by an explicit
|
|
`--override-tensor` regex (the expert `\.ffn_(up|down|gate|gate_up)_(ch|)exps`
|
|
pattern, buffer type `CPU`) and 7 CPU threads, native `draft-mtp`
|
|
speculative decoding (bundled MoE MTP head, n_max 4), f16 KV, context
|
|
688128 (3 slots of ~229K each, 32k headroom per stream), batch 4096 /
|
|
ubatch 2048, `--no-mmap`. The
|
|
unit is `scripts/frank-default-small.service`. It launches through
|
|
`scripts/serve-frank-small.sh`, which reads tunables from
|
|
`scripts/frank-default-small.env`. See [[dealign-ornith-crack]].
|
|
|
|
### Dankclaw — frank-default-large
|
|
|
|
The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
|
|
`kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` (Q6_0 AGENT, dense)
|
|
under the alias `frank-default-large` on port 7770. It runs
|
|
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
|
|
7.15 SDK in the retired ling3-ciru venv) with a standalone MTP Q4_0 draft
|
|
head (`--model-draft` and `--spec-type draft-mtp`), f16 KV, full 262144
|
|
context (1 slot. Concurrent requests queue; the server never preempts a
|
|
running generation), batch 4096 / ubatch 4096, 16 context checkpoints,
|
|
8 GiB prompt cache, and reasoning in deepseek format. The unit file is
|
|
`scripts/frank-default-large.service`. It launches through
|
|
`scripts/serve-frank-large.sh`, which reads tunables from
|
|
`scripts/frank-default-large.env`, applying the Qwen-Sharp chat template.
|
|
See [[kingjones-strix-mtp]].
|
|
|
|
> The kingjones Q6 AGENT model has served the slot since 2026-08-24; the
|
|
> v3 PromptForge payload is retired on this slot.
|
|
> Restored 2026-08-21. The intermediate DFlash2 Vulkan deployment was
|
|
> retired the same day. See [[dflash2-large-vulkan]] for the archived runbook.
|
|
|
|
## Validation
|
|
|
|
The deploy script checks the health endpoint after a restart. For deeper
|
|
checks, run a completion request against the endpoint:
|
|
|
|
```
|
|
curl -s http://127.0.0.1:7770/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model":"frank-default-small","messages":[{"role":"user","content":"Say OK"}]}'
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
If the health check fails after a deploy:
|
|
|
|
1. Read the service logs with `journalctl -u <service> -n 200`.
|
|
2. Confirm that the model files exist in the cache.
|
|
3. Confirm that the GPU has free memory (`nvidia-smi` on Dankserver, `rocm-smi` on Dankclaw).
|
|
4. Check the service status with `systemctl status <service>`.
|
|
|
|
## Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13)
|
|
|
|
On 2026-08-13 the old `frank-default-large` service crash-looped for hours
|
|
on Dankclaw. Root cause: the env files exported `TEMP=1.0` (a sampling
|
|
default). The ROCm HIP runtime reads `TEMP` as its JIT kernel-cache
|
|
directory, used the path `1.0`, failed, and segfaulted on a NULL object.
|
|
Fix: rename the sampling variable to `TEMPERATURE` (commit `227ce01`).
|
|
|
|
If a service segfaults at load with no obvious cause, compare the exported
|
|
env against the HIP runtime's env reads. Do not name any tunable `TEMP`,
|
|
`HOST`, or `GPU_MAX_HEAP_SIZE`.
|
|
|
|
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
|
|
|
|
Historical: on the retired topology, parallel jobs on the Dankserver 35B
|
|
service died with a cuBLAS crash under researcher load. The affected
|
|
service is retired. A client abort logs `W srv stop: cancel task,
|
|
id_task = N`. A natural stop logs only `slot release` with
|
|
`truncated = 0`. The server never preempts a running generation.
|
|
|
|
## Incident: frank-default-small Crash Loop After Distro Upgrade (2026-08-21)
|
|
|
|
On 2026-08-21 a release upgrade moved Dankserver from Ubuntu 24.04 to
|
|
Ubuntu 26.04 (Resolute Raccoon). The upgrade removed the CUDA 12 runtime
|
|
packages: `libcudart12`, `libcublas12`, `libcublaslt12`, and
|
|
`nvidia-cuda-toolkit`. The `llama-server` binary links
|
|
`libcudart.so.12` and `libcublas.so.12`. On every start the binary exited
|
|
with code 127. systemd restart-looped the service continuously. The health
|
|
endpoint on port 7770 stayed down.
|
|
|
|
Fix: reinstall the runtime packages. The same versions remain in the 26.04
|
|
repository.
|
|
|
|
sudo apt-get install -y libcudart12 libcublas12 libcublaslt12
|
|
|
|
Verify that the binary resolves all libraries:
|
|
|
|
ldd /home/dank/llama.cpp/build/bin/llama-server | grep "not found"
|
|
|
|
Start the service and check the health endpoint on port 7770.
|
|
|
|
After any release upgrade on Dankserver, run the `ldd` check above. A
|
|
missing CUDA 12 library means the packages need a reinstall.
|
|
|
|
## See Also
|
|
|
|
- [AGENTS.md](../../AGENTS.md) — the git workflow rules
|
|
- [README.md](../../README.md) — repository overview
|
|
- [index.md](index.md) — LLM fleet knowledge base index
|
|
- [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) — the full redo runbook
|
|
- [docs/index.md](../index.md) — the project knowledge base (podman stacks)
|