Frank
fbfbc5c6db
Operator request (2026-08-22): the primary inference server (frank-default-large, the 27B CIRU model on Dankclaw) moves to the standard llama.cpp port 7770. Both LLM services now listen on :7770, distinguished by host (dankserver vs dankclaw). Functional changes: - scripts/frank-default-large.env: LISTEN_PORT 52625 -> 7770 - scripts/serve-frank-large.sh: default LISTEN_PORT 52625 -> 7770 - scripts/deploy.sh: LLM_HEALTH[dankclaw] -> :7770 - open-webui quadlet: OPENAI_API_BASE_URL -> 192.168.0.143:7770 (re-pointed so chat keeps working; data dir preserved) - scripts/bench-fast-greedy.sh: PORT 52625 -> 7770 Tests updated to the new topology (both LLM endpoints on :7770 by host; open-webui -> :7770). Docs + pi KB updated; dated historical runbooks keep their point-in-time ports. models.json (pi client) re-pointed to :7770 separately, effective on the next client restart.
225 lines
8.6 KiB
Markdown
225 lines
8.6 KiB
Markdown
# Deployment and Update Process
|
|
|
|
This page describes how to deploy and update the LLM servers on Dankserver
|
|
and Dankclaw. The process uses git as the single source of truth. All script
|
|
edits flow through this repository.
|
|
|
|
> 2026-08-22: the former `vllmScripts` repository is merged into this repo.
|
|
> Both LLM services now run from the `/home/dank/serverAppConfigs` checkout
|
|
> on each host. The retired checkouts (`/home/dank/vLLM` on Dankserver,
|
|
> `/home/dank/vllmScripts` on Dankclaw) are archived on the hosts.
|
|
|
|
## Overview
|
|
|
|
Two servers run LLM inference services. Each server pulls its scripts from
|
|
this repository. The deployment process moves a committed change from the
|
|
repository to a running service.
|
|
|
|
| Target | Host | Runtime | Service | Port |
|
|
|--------|------|---------|---------|------|
|
|
| Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 |
|
|
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 7770 |
|
|
|
|
## Workflow
|
|
|
|
The deployment workflow has seven steps.
|
|
|
|
```mermaid
|
|
graph LR
|
|
A[Edit scripts locally] --> B[Commit]
|
|
B --> C[Push to origin<br>Gitea on Dankclaw]
|
|
C --> D[Pull on target<br>git pull]
|
|
D --> E[Restart podman stack]
|
|
E --> F[Relink + restart LLM unit]
|
|
F --> G[Verify health endpoints]
|
|
```
|
|
|
|
1. Fetch the current script from the target machine.
|
|
2. Commit the fetched version in this repository.
|
|
3. Edit the script locally in the working tree.
|
|
4. Run the tests (`./tests/run.sh` must pass).
|
|
5. Commit and push the change to origin.
|
|
6. Pull the change on the target machine.
|
|
7. Restart the services and verify that they are healthy.
|
|
|
|
## Deploy Script
|
|
|
|
The `scripts/deploy.sh` script is the unified deployer. One run per target
|
|
covers both workloads on that host: the podman stack and the LLM service.
|
|
Run it from the repository root.
|
|
|
|
Run this command to deploy to Dankserver:
|
|
|
|
```
|
|
./scripts/deploy.sh dankserver
|
|
```
|
|
|
|
Run this command to deploy to Dankclaw:
|
|
|
|
```
|
|
./scripts/deploy.sh dankclaw
|
|
```
|
|
|
|
Add the `--dry-run` flag to preview without making changes:
|
|
|
|
```
|
|
./scripts/deploy.sh dankserver --dry-run
|
|
```
|
|
|
|
The script does these actions:
|
|
|
|
1. Verifies that the local working tree has no uncommitted changes.
|
|
2. Pushes the `main` branch to origin.
|
|
3. Pulls the repository on the target machine.
|
|
4. Restarts the podman stack (Dankclaw only. The Dankserver media stack
|
|
runs as bare containers).
|
|
5. Relinks the LLM systemd units, reloads systemd, enables, and restarts
|
|
the active LLM service.
|
|
6. Waits for the Caddy homepage and the LLM health endpoint (over ssh).
|
|
|
|
Set `FRANK_SUDO_PW` when running headless on a host without passwordless
|
|
sudo, so that remote sudo reads the password from stdin (`sudo -S`). The
|
|
password is never stored in this repo. Both fleet hosts currently allow
|
|
passwordless sudo for the deploy user (verified 2026-08-22). The
|
|
passwordless restart helpers from the retired topology are obsolete and are
|
|
not used.
|
|
|
|
Health endpoints:
|
|
|
|
- Dankserver: `http://127.0.0.1:7770/health`
|
|
- Dankclaw: `http://127.0.0.1:7770/health`
|
|
|
|
## Manual Update
|
|
|
|
### Update Dankserver (frank-default-small)
|
|
|
|
Run these commands on Dankserver:
|
|
|
|
```
|
|
cd /home/dank/serverAppConfigs
|
|
git pull origin main
|
|
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart frank-default-small
|
|
curl -sf http://127.0.0.1:7770/health
|
|
```
|
|
|
|
### Update Dankclaw (frank-default-large)
|
|
|
|
Run these commands on Dankclaw:
|
|
|
|
```
|
|
cd /home/dank/serverAppConfigs
|
|
git pull origin main
|
|
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart frank-default-large
|
|
curl -sf http://127.0.0.1:7770/health
|
|
```
|
|
|
|
## Service Units
|
|
|
|
### Dankserver — frank-default-small
|
|
|
|
The active service on Dankserver is upstream llama.cpp (b10121) serving
|
|
`Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q6_K_P.gguf` under the
|
|
alias `frank-default-small` on port 7770. It runs the stock
|
|
`/home/dank/llama.cpp/build/bin/llama-server` binary with the 38 MoE
|
|
experts on CPU (`--n-cpu-moe 38`) and 7 CPU threads, ngram-mod
|
|
speculative decoding (n_min 48 / n_max 64 / n_match 24), f16 KV for both
|
|
K and V, context 786432 (3 slots of ~262K each), batch 4096 / ubatch 2048,
|
|
`--no-mmap`. The unit file is `scripts/frank-default-small.service`. It
|
|
launches through `scripts/serve-frank-small.sh`, which reads tunables from
|
|
`scripts/frank-default-small.env`.
|
|
|
|
### Dankclaw — frank-default-large
|
|
|
|
The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
|
|
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v3 five-file payload: GGUF
|
|
+ FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj) under the alias
|
|
`frank-default-large` on port 7770 (moved from 52625 on 2026-08-22). It runs
|
|
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
|
|
7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP
|
|
(`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context
|
|
(1 slot. Concurrent requests queue. The server never preempts a running
|
|
generation), batch 4096 / ubatch 4096, 16 context checkpoints, 8 GiB
|
|
prompt cache, and reasoning in deepseek format. The unit file is
|
|
`scripts/frank-default-large.service`. It launches through
|
|
`scripts/serve-frank-large.sh`, which reads tunables from
|
|
`scripts/frank-default-large.env`. See [[ciru-activefpx-promptforge]].
|
|
|
|
> Restored 2026-08-21. The intermediate DFlash2 Vulkan deployment was
|
|
> retired the same day. See [[dflash2-large-vulkan]] for the archived runbook.
|
|
|
|
## Validation
|
|
|
|
The deploy script checks the health endpoint after a restart. For deeper
|
|
checks, run a completion request against the endpoint:
|
|
|
|
```
|
|
curl -s http://127.0.0.1:7770/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model":"frank-default-small","messages":[{"role":"user","content":"Say OK"}]}'
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
If the health check fails after a deploy:
|
|
|
|
1. Read the service logs with `journalctl -u <service> -n 200`.
|
|
2. Confirm that the model files exist in the cache.
|
|
3. Confirm that the GPU has free memory (`nvidia-smi` on Dankserver, `rocm-smi` on Dankclaw).
|
|
4. Check the service status with `systemctl status <service>`.
|
|
|
|
## Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13)
|
|
|
|
On 2026-08-13 the old `frank-default-large` service crash-looped for hours
|
|
on Dankclaw. Root cause: the env files exported `TEMP=1.0` (a sampling
|
|
default). The ROCm HIP runtime reads `TEMP` as its JIT kernel-cache
|
|
directory, used the path `1.0`, failed, and segfaulted on a NULL object.
|
|
Fix: rename the sampling variable to `TEMPERATURE` (commit `227ce01`).
|
|
|
|
If a service segfaults at load with no obvious cause, compare the exported
|
|
env against the HIP runtime's env reads. Do not name any tunable `TEMP`,
|
|
`HOST`, or `GPU_MAX_HEAP_SIZE`.
|
|
|
|
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
|
|
|
|
Historical: on the retired topology, parallel jobs on the Dankserver 35B
|
|
service died with a cuBLAS crash under researcher load. The affected
|
|
service is retired. A client abort logs `W srv stop: cancel task,
|
|
id_task = N`. A natural stop logs only `slot release` with
|
|
`truncated = 0`. The server never preempts a running generation.
|
|
|
|
## Incident: frank-default-small Crash Loop After Distro Upgrade (2026-08-21)
|
|
|
|
On 2026-08-21 a release upgrade moved Dankserver from Ubuntu 24.04 to
|
|
Ubuntu 26.04 (Resolute Raccoon). The upgrade removed the CUDA 12 runtime
|
|
packages: `libcudart12`, `libcublas12`, `libcublaslt12`, and
|
|
`nvidia-cuda-toolkit`. The `llama-server` binary links
|
|
`libcudart.so.12` and `libcublas.so.12`. On every start the binary exited
|
|
with code 127. systemd restart-looped the service continuously. The health
|
|
endpoint on port 7770 stayed down.
|
|
|
|
Fix: reinstall the runtime packages. The same versions remain in the 26.04
|
|
repository.
|
|
|
|
sudo apt-get install -y libcudart12 libcublas12 libcublaslt12
|
|
|
|
Verify that the binary resolves all libraries:
|
|
|
|
ldd /home/dank/llama.cpp/build/bin/llama-server | grep "not found"
|
|
|
|
Start the service and check the health endpoint on port 7770.
|
|
|
|
After any release upgrade on Dankserver, run the `ldd` check above. A
|
|
missing CUDA 12 library means the packages need a reinstall.
|
|
|
|
## See Also
|
|
|
|
- [AGENTS.md](../../AGENTS.md) — the git workflow rules
|
|
- [README.md](../../README.md) — repository overview
|
|
- [index.md](index.md) — LLM fleet knowledge base index
|
|
- [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) — the full redo runbook
|
|
- [docs/index.md](../index.md) — the project knowledge base (podman stacks)
|