Files

T

Frank af6ffeeb64 Switch frank-default-small to Ornith Q6_K

2026-08-25 00:29:14 -07:00

8.9 KiB

Raw Blame History

Deployment and Update Process

This page describes how to deploy and update the LLM servers on Dankserver and Dankclaw. The process uses git as the single source of truth. All script edits flow through this repository.

2026-08-22: the former vllmScripts repository is merged into this repo. Both LLM services now run from the /home/dank/serverAppConfigs checkout on each host. The retired checkouts (/home/dank/vLLM on Dankserver, /home/dank/vllmScripts on Dankclaw) are archived on the hosts.

Overview

Two servers run LLM inference services. Each server pulls its scripts from this repository. The deployment process moves a committed change from the repository to a running service.

Target	Host	Runtime	Service	Port
Dankserver	dank@dankserver	upstream llama.cpp (Ornith-1.5-35B-A3B-CRACK Q6_K, native MTP draft, selected MoE experts offloaded via --override-tensor)	frank-default-small	7770
Dankclaw	dank@dankclaw	CIRU ROCmFPX llama.cpp (Qwen3.8-27B-Uncensored-ROCmFP4 Q6 AGENT, standalone MTP Q4_0 head, 262K ctx / 1 slot)	frank-default-large	7770

Workflow

The deployment workflow has seven steps.

graph LR
    A[Edit scripts locally] --> B[Commit]
    B --> C[Push to origin<br>Gitea on Dankclaw]
    C --> D[Pull on target<br>git pull]
    D --> E[Restart podman stack]
    E --> F[Relink + restart LLM unit]
    F --> G[Verify health endpoints]

Fetch the current script from the target machine.
Commit the fetched version in this repository.
Edit the script locally in the working tree.
Run the tests (./tests/run.sh must pass).
Commit and push the change to origin.
Pull the change on the target machine.
Restart the services and verify that they are healthy.

Deploy Script

The scripts/deploy.sh script is the unified deployer. One run per target covers both workloads on that host: the podman stack and the LLM service. Run it from the repository root.

Run this command to deploy to Dankserver:

./scripts/deploy.sh dankserver

Run this command to deploy to Dankclaw:

./scripts/deploy.sh dankclaw

Add the --dry-run flag to preview without making changes:

./scripts/deploy.sh dankserver --dry-run

The script does these actions:

Verifies that the local working tree has no uncommitted changes.
Pushes the main branch to origin.
Pulls the repository on the target machine.
Restarts the podman stack (Dankclaw only. The Dankserver media stack runs as bare containers).
Relinks the LLM systemd units, reloads systemd, enables, and restarts the active LLM service.
Waits for the Caddy homepage and the LLM health endpoint (over ssh).

Set FRANK_SUDO_PW when running headless on a host without passwordless sudo, so that remote sudo reads the password from stdin (sudo -S). The password is never stored in this repo. Both fleet hosts currently allow passwordless sudo for the deploy user (verified 2026-08-22). The passwordless restart helpers from the retired topology are obsolete and are not used.

Health endpoints:

Dankserver: http://127.0.0.1:7770/health
Dankclaw: http://127.0.0.1:7770/health

Manual Update

Update Dankserver (frank-default-small)

Run these commands on Dankserver:

cd /home/dank/serverAppConfigs
git pull origin main
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-small
curl -sf http://127.0.0.1:7770/health

Update Dankclaw (frank-default-large)

Run these commands on Dankclaw:

cd /home/dank/serverAppConfigs
git pull origin main
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:7770/health

Service Units

Dankserver — frank-default-small

The active service on Dankserver is upstream llama.cpp (b10121) serving Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf under the alias frank-default-small on port 7770. It runs the stock /home/dank/llama.cpp/build/bin/llama-server binary with expert tensors from blocks 0 through 36 pinned to CPU by an explicit --override-tensor regex. Expert tensors from blocks 37 through 40 stay on CUDA. The service uses 7 CPU threads, native draft-mtp speculative decoding, f16 KV, context 589824 (3 slots of 196608 each), batch 4096, ubatch 2048, and --no-mmap. The unit is scripts/frank-default-small.service. It launches through scripts/serve-frank-small.sh, which reads tunables from scripts/frank-default-small.env. The chat template comes from peculiar-ragdoll/Qwen-Sharp-Chat-Templates. See dealign-ornith-crack.

Dankclaw — frank-default-large

The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP (Q6_0 AGENT, dense) under the alias frank-default-large on port 7770. It runs /home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server (ROCm 7.15 SDK in the retired ling3-ciru venv) with a standalone MTP Q4_0 draft head (--model-draft and --spec-type draft-mtp), f16 KV, full 262144 context (1 slot. Concurrent requests queue; the server never preempts a running generation), batch 4096 / ubatch 4096, 16 context checkpoints, 8 GiB prompt cache, and reasoning in deepseek format. The unit file is scripts/frank-default-large.service. It launches through scripts/serve-frank-large.sh, which reads tunables from scripts/frank-default-large.env, applying the Qwen-Sharp chat template. See kingjones-strix-mtp.

The kingjones Q6 AGENT model has served the slot since 2026-08-24; the v3 PromptForge payload is retired on this slot. Restored 2026-08-21. The intermediate DFlash2 Vulkan deployment was retired the same day. See dflash2-large-vulkan for the archived runbook.

Validation

The deploy script checks the health endpoint after a restart. For deeper checks, run a completion request against the endpoint:

curl -s http://127.0.0.1:7770/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"frank-default-small","messages":[{"role":"user","content":"Say OK"}]}'

Troubleshooting

If the health check fails after a deploy:

Read the service logs with journalctl -u <service> -n 200.
Confirm that the model files exist in the cache.
Confirm that the GPU has free memory (nvidia-smi on Dankserver, rocm-smi on Dankclaw).
Check the service status with systemctl status <service>.

Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13)

On 2026-08-13 the old frank-default-large service crash-looped for hours on Dankclaw. Root cause: the env files exported TEMP=1.0 (a sampling default). The ROCm HIP runtime reads TEMP as its JIT kernel-cache directory, used the path 1.0, failed, and segfaulted on a NULL object. Fix: rename the sampling variable to TEMPERATURE (commit 227ce01).

If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Do not name any tunable TEMP, HOST, or GPU_MAX_HEAP_SIZE.

Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)

Historical: on the retired topology, parallel jobs on the Dankserver 35B service died with a cuBLAS crash under researcher load. The affected service is retired. A client abort logs W srv stop: cancel task, id_task = N. A natural stop logs only slot release with truncated = 0. The server never preempts a running generation.

Incident: frank-default-small Crash Loop After Distro Upgrade (2026-08-21)

On 2026-08-21 a release upgrade moved Dankserver from Ubuntu 24.04 to Ubuntu 26.04 (Resolute Raccoon). The upgrade removed the CUDA 12 runtime packages: libcudart12, libcublas12, libcublaslt12, and nvidia-cuda-toolkit. The llama-server binary links libcudart.so.12 and libcublas.so.12. On every start the binary exited with code 127. systemd restart-looped the service continuously. The health endpoint on port 7770 stayed down.

Fix: reinstall the runtime packages. The same versions remain in the 26.04 repository.

sudo apt-get install -y libcudart12 libcublas12 libcublaslt12

Verify that the binary resolves all libraries:

ldd /home/dank/llama.cpp/build/bin/llama-server | grep "not found"

Start the service and check the health endpoint on port 7770.

After any release upgrade on Dankserver, run the ldd check above. A missing CUDA 12 library means the packages need a reinstall.

8.9 KiB Raw Blame History