Frank
64cdfc8529
- Dankclaw System Specs: correct RAM to 64 GiB (61 visible), GPU budget to the rocm-smi 64 GiB SVM figure, and replace the vLLM relevance section with the CIRU ROCmFPX runtime facts. Note the retired vLLM venvs in .AGENTTRASH. Refresh the disk figures. - deployment.md: update the dankclaw target row (CIRU runtime).
171 lines
6.9 KiB
Markdown
171 lines
6.9 KiB
Markdown
# Deployment and Update Process
|
|
|
|
This page describes how to deploy and update the LLM servers on Dankserver and Dankclaw. The process uses git as the single source of truth. All script edits flow through this repository.
|
|
|
|
## Overview
|
|
|
|
Two servers run LLM inference services. Each server pulls its scripts from this repository. The deployment process moves a committed change from the repository to a running service.
|
|
|
|
| Target | Host | Runtime | Service | Port |
|
|
|--------|------|---------|---------|------|
|
|
| Dankserver | dank@dankserver | llama.cpp FastMTP (Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod) | frank-default-large | 7770 |
|
|
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView) | frank-default-small | 52625 |
|
|
|
|
## Workflow
|
|
|
|
The deployment workflow has six steps.
|
|
|
|
```mermaid
|
|
graph LR
|
|
A[Edit scripts locally] --> B[Commit]
|
|
B --> C[Push to origin<br>Gitea on Dankclaw]
|
|
C --> D[Pull on target<br>git pull]
|
|
D --> E[Restart systemd service]
|
|
E --> F[Verify health endpoint]
|
|
```
|
|
|
|
1. Fetch the current script from the target machine.
|
|
2. Commit the fetched version in this repository.
|
|
3. Edit the script locally in the working tree.
|
|
4. Commit and push the change to origin.
|
|
5. Pull the change on the target machine.
|
|
6. Restart the service and verify that it is healthy.
|
|
|
|
## Deploy Script
|
|
|
|
The `scripts/deploy.sh` script executes steps 4 through 6. Run it from the repository root.
|
|
|
|
Run this command to deploy to Dankserver:
|
|
|
|
```
|
|
FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver
|
|
```
|
|
|
|
Run this command to deploy to Dankclaw:
|
|
|
|
```
|
|
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw
|
|
```
|
|
|
|
Add the `--dry-run` flag to preview without making changes:
|
|
|
|
```
|
|
./scripts/deploy.sh dankserver --dry-run
|
|
```
|
|
|
|
The script does these actions:
|
|
|
|
1. Verifies that the local working tree has no uncommitted changes.
|
|
2. Pushes the `main` branch to origin.
|
|
3. Pulls the repository on the target machine.
|
|
4. Symlinks the systemd unit, reloads systemd, enables, and restarts the service.
|
|
5. Waits for the health endpoint to respond (over ssh).
|
|
|
|
Set `FRANK_SUDO_PW` when running headless, so that remote sudo reads the
|
|
password from stdin (`sudo -S`). The passwordless restart helpers from the
|
|
retired topology are obsolete and are not used.
|
|
|
|
Health endpoints:
|
|
|
|
- Dankserver: `http://127.0.0.1:7770/health`
|
|
- Dankclaw: `http://127.0.0.1:52625/health`
|
|
|
|
## Manual Update
|
|
|
|
### Update Dankserver (frank-default-large)
|
|
|
|
Run these commands on Dankserver:
|
|
|
|
```
|
|
cd /home/dank/vLLM
|
|
git pull origin main
|
|
sudo ln -sf /home/dank/vLLM/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart frank-default-large
|
|
curl -sf http://127.0.0.1:7770/health
|
|
```
|
|
|
|
### Update Dankclaw (frank-default-small)
|
|
|
|
Run these commands on Dankclaw:
|
|
|
|
```
|
|
cd /home/dank/vllmScripts
|
|
git pull origin main
|
|
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart frank-default-small
|
|
curl -sf http://127.0.0.1:52625/health
|
|
```
|
|
|
|
## Service Units
|
|
|
|
### Dankserver — frank-default-large
|
|
|
|
The active service on Dankserver is llama.cpp serving
|
|
`Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-IQ4_XS.gguf` under the alias
|
|
`frank-default-large` on port 7770. It runs the patched FastMTP build
|
|
(`/home/dank/llama.cpp-fastmtp/build/bin/llama-server`, commit 4df29be +
|
|
the HauhauCS patch) with the FastMTP 32K draft sidecar (`--spec-type
|
|
draft-mtp`, `--spec-draft-n-max 2`, `--spec-draft-ngl all`) and the
|
|
ngram-mod fallback. Context 131072 (128K, single slot), KV K q8_0 + V f16,
|
|
batch 2048 / ubatch 1024, `--no-mmap`, full GPU offload. The unit file is
|
|
`scripts/frank-default-large.service`. It launches through
|
|
`scripts/serve-frank-large.sh`, which reads tunables from
|
|
`scripts/frank-default-large.env`.
|
|
|
|
### Dankclaw — frank-default-small
|
|
|
|
The active service on Dankclaw is vLLM (ROCm) serving
|
|
`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` under the served model name
|
|
`frank-default-small` on port 52625. Full 262144 context, unquantized KV
|
|
cache, `--max-num-seqs 4`, `--max-num-batched-tokens 8192`, ngram
|
|
speculative decoding, `--reasoning-parser qwen3`, tool calling
|
|
(`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with the custom
|
|
chat template `/home/dank/models/qwen36-chat_template-tools.jinja`), and
|
|
prefix caching (`--enable-prefix-caching`). The model is hybrid (GatedDeltaNet
|
|
+ full attention); vLLM keeps prefix caching opt-in for hybrid models, so the
|
|
flag is required. It reuses the KV of repeated prompt prefixes (pi re-sends
|
|
the same system prompt + tool schema every turn), cutting follow-up TTFT
|
|
roughly 4x (2023-token prefix: 4.6s cold -> 1.1s cached, 2026-08-17). The
|
|
unit file is `scripts/frank-default-small.service`. It launches through
|
|
`scripts/serve-frank-small.sh`, which reads tunables from
|
|
`scripts/frank-default-small.env`.
|
|
|
|
## Validation
|
|
|
|
The deploy script checks the health endpoint after a restart. For deeper checks, run a completion request against the endpoint:
|
|
|
|
```
|
|
curl -s http://127.0.0.1:7770/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model":"frank-default-large","messages":[{"role":"user","content":"Say OK"}]}'
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
If the health check fails after a deploy:
|
|
|
|
1. Read the service logs with `journalctl -u <service> -n 200`.
|
|
2. Confirm that the model files exist in the cache.
|
|
3. Confirm that the GPU has free memory (`nvidia-smi` on Dankserver, `rocm-smi` on Dankclaw).
|
|
4. Check the service status with `systemctl status <service>`.
|
|
|
|
## Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13)
|
|
|
|
On 2026-08-13 the old `frank-default-large` service crash-looped for hours on Dankclaw. Root cause: the env files exported `TEMP=1.0` (a sampling default). The ROCm HIP runtime reads `TEMP` as its JIT kernel-cache directory, used the path `1.0`, failed, and segfaulted on a NULL object. Fix: rename the sampling variable to `TEMPERATURE` (commit `227ce01`).
|
|
|
|
If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Do not name any tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`.
|
|
|
|
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
|
|
|
|
Historical: on the retired topology, parallel jobs on the Dankserver 35B service died with a cuBLAS crash under researcher load. The affected service is retired. A client abort logs `W srv stop: cancel task, id_task = N`; a natural stop logs only `slot release` with `truncated = 0`. The server never preempts a running generation.
|
|
|
|
## See Also
|
|
|
|
- [AGENTS.md](../AGENTS.md) — the git workflow rules
|
|
- [README.md](../README.md) — repository overview
|
|
- [index.md](index.md) — knowledge base index
|
|
- [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) — the full redo runbook
|
|
- [serverAppConfigs knowledge base](../serverAppConfigs/docs/index.md) —
|
|
the sibling Podman app stack on the same hosts |