dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: one knowledge base for both workloads

Browse Source 
- docs/index.md: LLM Fleet section (all 34 pages linked), merge note,
  See Also now internal
- docs/LLM/index.md: retitled, dangling [[Dankserver]]/[[Dankclaw]]
  links fixed, sibling section replaced with internal links
- docs/LLM/deployment.md: rewritten for the unified deploy flow,
  manual-update paths point at the merged checkout, incident history
  kept intact
- Deployment Flow / Dankclaw Stack / Gitea: stale vllmScripts
  references and retired-topology notes updated to the 2026-08-22
  state
- Historical docs (frank-redo, PP deployment, FastFlowLM) keep their
  dated runbook paths as-is (records of retired topologies)
This commit is contained in:
Frank committed 2026-08-22 15:50:39 -07:00
1 parent 0461988884
commit e367f0f43c
6 files changed
+177 -68
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/LLM/deployment.md
+73 -35
View File
@@ -1,10 +1,19 @@
# Deployment and Update Process
This page describes how to deploy and update the LLM servers on Dankserver and Dankclaw. The process uses git as the single source of truth. All script edits flow through this repository.
This page describes how to deploy and update the LLM servers on Dankserver
and Dankclaw. The process uses git as the single source of truth. All script
edits flow through this repository.
> 2026-08-22: the former `vllmScripts` repository is merged into this repo.
> Both LLM services now run from the `/home/dank/serverAppConfigs` checkout
> on each host. The retired checkouts (`/home/dank/vLLM` on Dankserver,
> `/home/dank/vllmScripts` on Dankclaw) are archived on the hosts.
## Overview
Two servers run LLM inference services. Each server pulls its scripts from this repository. The deployment process moves a committed change from the repository to a running service.
Two servers run LLM inference services. Each server pulls its scripts from
this repository. The deployment process moves a committed change from the
repository to a running service.
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
@@ -13,38 +22,42 @@ Two servers run LLM inference services. Each server pulls its scripts from this
## Workflow
The deployment workflow has six steps.
The deployment workflow has seven steps.
```mermaid
graph LR
A[Edit scripts locally] --> B[Commit]
B --> C[Push to origin<br>Gitea on Dankclaw]
C --> D[Pull on target<br>git pull]
D --> E[Restart systemd service]
E --> F[Verify health endpoint]
D --> E[Restart podman stack]
E --> F[Relink + restart LLM unit]
F --> G[Verify health endpoints]
```
1. Fetch the current script from the target machine.
2. Commit the fetched version in this repository.
3. Edit the script locally in the working tree.
4. Commit and push the change to origin.
5. Pull the change on the target machine.
6. Restart the service and verify that it is healthy.
4. Run the tests (`./tests/run.sh` must pass).
5. Commit and push the change to origin.
6. Pull the change on the target machine.
7. Restart the services and verify that they are healthy.
## Deploy Script
The `scripts/deploy.sh` script executes steps 4 through 6. Run it from the repository root.
The `scripts/deploy.sh` script is the unified deployer. One run per target
covers both workloads on that host: the podman stack and the LLM service.
Run it from the repository root.
Run this command to deploy to Dankserver:
```
FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver
./scripts/deploy.sh dankserver
```
Run this command to deploy to Dankclaw:
```
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankclaw
```
Add the `--dry-run` flag to preview without making changes:
@@ -58,12 +71,18 @@ The script does these actions:
1. Verifies that the local working tree has no uncommitted changes.
2. Pushes the `main` branch to origin.
3. Pulls the repository on the target machine.
4. Symlinks the systemd unit, reloads systemd, enables, and restarts the service.
5. Waits for the health endpoint to respond (over ssh).
4. Restarts the podman stack (Dankclaw only. The Dankserver media stack
runs as bare containers).
5. Relinks the LLM systemd units, reloads systemd, enables, and restarts
the active LLM service.
6. Waits for the Caddy homepage and the LLM health endpoint (over ssh).
Set `FRANK_SUDO_PW` when running headless, so that remote sudo reads the
password from stdin (`sudo -S`). The passwordless restart helpers from the
retired topology are obsolete and are not used.
Set `FRANK_SUDO_PW` when running headless on a host without passwordless
sudo, so that remote sudo reads the password from stdin (`sudo -S`). The
password is never stored in this repo. Both fleet hosts currently allow
passwordless sudo for the deploy user (verified 2026-08-22). The
passwordless restart helpers from the retired topology are obsolete and are
not used.
Health endpoints:
@@ -77,9 +96,9 @@ Health endpoints:
Run these commands on Dankserver:
```
cd /home/dank/vLLM
cd /home/dank/serverAppConfigs
git pull origin main
sudo ln -sf /home/dank/vLLM/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-small
curl -sf http://127.0.0.1:7770/health
@@ -90,9 +109,9 @@ curl -sf http://127.0.0.1:7770/health
Run these commands on Dankclaw:
```
cd /home/dank/vllmScripts
cd /home/dank/serverAppConfigs
git pull origin main
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:52625/health
@@ -117,11 +136,12 @@ launches through `scripts/serve-frank-small.sh`, which reads tunables from
The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v3 five-file payload: GGUF
+ FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj) under the alias `frank-default-large` on port 52625. It runs
+ FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj) under the alias
`frank-default-large` on port 52625. It runs
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP
(`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context
(1 slot; concurrent requests queue, the server never preempts a running
(1 slot. Concurrent requests queue. The server never preempts a running
generation), batch 4096 / ubatch 4096, 16 context checkpoints, 8 GiB
prompt cache, and reasoning in deepseek format. The unit file is
`scripts/frank-default-large.service`. It launches through
@@ -129,11 +149,12 @@ prompt cache, and reasoning in deepseek format. The unit file is
`scripts/frank-default-large.env`. See [[ciru-activefpx-promptforge]].
> Restored 2026-08-21. The intermediate DFlash2 Vulkan deployment was
> retired the same day; see [[dflash2-large-vulkan]] for the archived runbook.
> retired the same day. See [[dflash2-large-vulkan]] for the archived runbook.
## Validation
The deploy script checks the health endpoint after a restart. For deeper checks, run a completion request against the endpoint:
The deploy script checks the health endpoint after a restart. For deeper
checks, run a completion request against the endpoint:
```
curl -s http://127.0.0.1:7770/v1/chat/completions \
@@ -152,19 +173,36 @@ If the health check fails after a deploy:
## Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13)
On 2026-08-13 the old `frank-default-large` service crash-looped for hours on Dankclaw. Root cause: the env files exported `TEMP=1.0` (a sampling default). The ROCm HIP runtime reads `TEMP` as its JIT kernel-cache directory, used the path `1.0`, failed, and segfaulted on a NULL object. Fix: rename the sampling variable to `TEMPERATURE` (commit `227ce01`).
On 2026-08-13 the old `frank-default-large` service crash-looped for hours
on Dankclaw. Root cause: the env files exported `TEMP=1.0` (a sampling
default). The ROCm HIP runtime reads `TEMP` as its JIT kernel-cache
directory, used the path `1.0`, failed, and segfaulted on a NULL object.
Fix: rename the sampling variable to `TEMPERATURE` (commit `227ce01`).
If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Do not name any tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`.
If a service segfaults at load with no obvious cause, compare the exported
env against the HIP runtime's env reads. Do not name any tunable `TEMP`,
`HOST`, or `GPU_MAX_HEAP_SIZE`.
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
Historical: on the retired topology, parallel jobs on the Dankserver 35B service died with a cuBLAS crash under researcher load. The affected service is retired. A client abort logs `W srv stop: cancel task, id_task = N`; a natural stop logs only `slot release` with `truncated = 0`. The server never preempts a running generation.
Historical: on the retired topology, parallel jobs on the Dankserver 35B
service died with a cuBLAS crash under researcher load. The affected
service is retired. A client abort logs `W srv stop: cancel task,
id_task = N`. A natural stop logs only `slot release` with
`truncated = 0`. The server never preempts a running generation.
## Incident: frank-default-small Crash Loop After Distro Upgrade (2026-08-21)
On 2026-08-21 a release upgrade moved Dankserver from Ubuntu 24.04 to Ubuntu 26.04 (Resolute Raccoon). The upgrade removed the CUDA 12 runtime packages: `libcudart12`, `libcublas12`, `libcublaslt12`, and `nvidia-cuda-toolkit`. The `llama-server` binary links `libcudart.so.12` and `libcublas.so.12`. On every start the binary exited with code 127. systemd restart-looped the service continuously. The health endpoint on port 7770 stayed down.
On 2026-08-21 a release upgrade moved Dankserver from Ubuntu 24.04 to
Ubuntu 26.04 (Resolute Raccoon). The upgrade removed the CUDA 12 runtime
packages: `libcudart12`, `libcublas12`, `libcublaslt12`, and
`nvidia-cuda-toolkit`. The `llama-server` binary links
`libcudart.so.12` and `libcublas.so.12`. On every start the binary exited
with code 127. systemd restart-looped the service continuously. The health
endpoint on port 7770 stayed down.
Fix: reinstall the runtime packages. The same versions remain in the 26.04 repository.
Fix: reinstall the runtime packages. The same versions remain in the 26.04
repository.
sudo apt-get install -y libcudart12 libcublas12 libcublaslt12
@@ -174,13 +212,13 @@ Verify that the binary resolves all libraries:
Start the service and check the health endpoint on port 7770.
After any release upgrade on Dankserver, run the `ldd` check above. A missing CUDA 12 library means the packages need a reinstall.
After any release upgrade on Dankserver, run the `ldd` check above. A
missing CUDA 12 library means the packages need a reinstall.
## See Also
- [AGENTS.md](../AGENTS.md) — the git workflow rules
- [README.md](../README.md) — repository overview
- [index.md](index.md) — knowledge base index
- [AGENTS.md](../../AGENTS.md) — the git workflow rules
- [README.md](../../README.md) — repository overview
- [index.md](index.md) — LLM fleet knowledge base index
- [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) — the full redo runbook
- [serverAppConfigs knowledge base](../serverAppConfigs/docs/index.md) —
the sibling Podman app stack on the same hosts
- [docs/index.md](../index.md) — the project knowledge base (podman stacks)
docs/LLM/index.md
+16 -9
View File
@@ -1,10 +1,15 @@
# vLLM Scripts — Knowledge Base
# LLM Fleet — Knowledge Base
Documentation for the LLM inference fleet on two local machines
(2026-08-22): upstream llama.cpp on Dankserver (frank-default-small) and
CIRU ROCmFPX llama.cpp with the PromptForge v3 runtime on Dankclaw
(frank-default-large).
> 2026-08-22: the former `vllmScripts` repository is merged into the
> serverAppConfigs repository. This knowledge base moved from the old repo's
> `docs/` root to `docs/LLM/`. The fleet scripts and units now live in the
> repo's `scripts/` directory.
## Deployment Topology
```mermaid
@@ -43,11 +48,11 @@ frank-default-large, port 52625]
- [[deployment]] — deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[ciru-activefpx-promptforge]] — the v3 CIRU PromptForge runtime on
Dankclaw (frank-default-large, active; restored 2026-08-22)
Dankclaw (frank-default-large, active. Restored 2026-08-22)
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
(deployed and reverted 2026-08-22)
- [[Dankserver]] — the upstream llama.cpp box (frank-default-small)
- [[Dankclaw]] — the CIRU ROCmFPX box (frank-default-large)
- [[Dankserver System Specs]] — the upstream llama.cpp box (frank-default-small)
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box (frank-default-large)
## CIRU Reference (Dankclaw)
@@ -88,10 +93,12 @@ frank-default-large, port 52625]
## Pipeline Parallelism (archive)
- [[Pipeline Parallelism]] — index of multi-node inference research. The PP
pair is retired; these pages are research history.
pair is retired. These pages are research history.
## Sibling projects
## Related
- [serverAppConfigs](../serverAppConfigs/docs/index.md) — the Podman app
stack on the same two hosts. Its OpenWebUI consumes the `:7770` endpoint
on Dankserver; its Gitea instance hosts this repository.
- [Project knowledge base](../index.md) — the Podman app stacks on the same
two hosts. OpenWebUI consumes the `:7770` endpoint on Dankserver. The
Gitea instance on Dankclaw hosts this repository.
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the unified
git-based deploy for both workloads