docs: one knowledge base for both workloads
- docs/index.md: LLM Fleet section (all 34 pages linked), merge note, See Also now internal - docs/LLM/index.md: retitled, dangling [[Dankserver]]/[[Dankclaw]] links fixed, sibling section replaced with internal links - docs/LLM/deployment.md: rewritten for the unified deploy flow, manual-update paths point at the merged checkout, incident history kept intact - Deployment Flow / Dankclaw Stack / Gitea: stale vllmScripts references and retired-topology notes updated to the 2026-08-22 state - Historical docs (frank-redo, PP deployment, FastFlowLM) keep their dated runbook paths as-is (records of retired topologies)
This commit is contained in:
1 parent
0461988884
commit
e367f0f43c
6 files changed
+177
-68
No files matched your search
@@ -1,6 +1,8 @@
|
|||||||
# Deployment Flow
|
# Deployment Flow
|
||||||
|
|
||||||
Deployment mirrors the `vllmScripts` repository: git is the single source of truth, and a `deploy.sh` script moves committed changes to the target hosts over SSH.
|
Git is the single source of truth. A `deploy.sh` script moves committed
|
||||||
|
changes to the target hosts over SSH. One run per target covers both
|
||||||
|
workloads on that host: the podman stack and the LLM service.
|
||||||
|
|
||||||
## Workflow
|
## Workflow
|
||||||
|
|
||||||
@@ -16,8 +18,9 @@ graph LR
|
|||||||
1. Edit files locally in this repository.
|
1. Edit files locally in this repository.
|
||||||
2. Commit and push.
|
2. Commit and push.
|
||||||
3. Run `./scripts/deploy.sh <dankclaw|dankserver>`.
|
3. Run `./scripts/deploy.sh <dankclaw|dankserver>`.
|
||||||
4. The script pulls the repo on the target and restarts the podman stack.
|
4. The script pulls the repo on the target and restarts the podman stack
|
||||||
5. It verifies service health after restart.
|
and the LLM service.
|
||||||
|
5. It verifies service health after restart (homepage and LLM endpoint).
|
||||||
|
|
||||||
## Deploy Script
|
## Deploy Script
|
||||||
|
|
||||||
@@ -54,4 +57,4 @@ The suite validates compose YAML, env consistency, Caddyfiles, docs completeness
|
|||||||
|
|
||||||
- [[Storage Policy]]
|
- [[Storage Policy]]
|
||||||
- [[Testing Policy]]
|
- [[Testing Policy]]
|
||||||
- [vllmScripts deployment docs](../../../vllmScripts/docs/deployment.md) — the model this follows
|
- [LLM deployment docs](../LLM/deployment.md) — the LLM half of the same flow
|
||||||
@@ -1,8 +1,8 @@
|
|||||||
# Dankclaw Stack
|
# Dankclaw Stack
|
||||||
|
|
||||||
Dankclaw (192.168.0.143) runs the AI, search, and development stack. It
|
Dankclaw (192.168.0.143) runs the AI, search, and development stack. It
|
||||||
hosts the vLLM inference endpoint (`frank-default-large`, `:52625`, from
|
hosts the LLM inference endpoint (`frank-default-large`, `:52625`, CIRU
|
||||||
the `vllmScripts` repo), the agent tools, the privacy search, and the
|
ROCmFPX llama.cpp), the agent tools, the privacy search, and the
|
||||||
observability stack. The llama.cpp endpoint (`frank-default-small`,
|
observability stack. The llama.cpp endpoint (`frank-default-small`,
|
||||||
`:7770`) runs on Dankserver (192.168.0.235).
|
`:7770`) runs on Dankserver (192.168.0.235).
|
||||||
|
|
||||||
@@ -17,7 +17,7 @@ graph LR
|
|||||||
S --> V[Gluetun VPN]
|
S --> V[Gluetun VPN]
|
||||||
O --> L[llama.cpp dankserver :7770]
|
O --> L[llama.cpp dankserver :7770]
|
||||||
O --> H[Hermes]
|
O --> H[Hermes]
|
||||||
O --> E[vLLM dankclaw :52625]
|
O --> E[CIRU llama.cpp :52625]
|
||||||
```
|
```
|
||||||
|
|
||||||
## Services
|
## Services
|
||||||
@@ -32,14 +32,14 @@ graph LR
|
|||||||
| Grafana | grafana.dankclaw.local | 3002 | [[Grafana]] |
|
| Grafana | grafana.dankclaw.local | 3002 | [[Grafana]] |
|
||||||
| Prometheus | prometheus.dankclaw.local | 9090 | [[Prometheus]] |
|
| Prometheus | prometheus.dankclaw.local | 9090 | [[Prometheus]] |
|
||||||
| Homepage | dankclaw.local | 80/443 | — |
|
| Homepage | dankclaw.local | 80/443 | — |
|
||||||
| frank-default-large (host service, Dankclaw) | — | 52625 | [vllmScripts](../../vllmScripts/docs/index.md) |
|
| frank-default-large (host service, Dankclaw) | — | 52625 | [[deployment]] |
|
||||||
| frank-default-small (host service, Dankserver) | — | 7770 | [vllmScripts](../../vllmScripts/docs/index.md) |
|
| frank-default-small (host service, Dankserver) | — | 7770 | [[deployment]] |
|
||||||
|
|
||||||
The last two rows are bare-metal systemd services, not containers. The
|
The last two rows are bare-metal systemd services, not containers. This
|
||||||
`vllmScripts` repository owns their scripts and knowledge base. The live
|
repo owns their scripts, units, and knowledge base (see
|
||||||
mapping (verified 2026-08-19) is the reverse of the 2026-08-17 note:
|
[docs/LLM/index.md](../LLM/index.md)). Current mapping (2026-08-22):
|
||||||
`frank-default-large` runs vLLM on Dankclaw; `frank-default-small` runs
|
`frank-default-large` runs the CIRU ROCmFPX llama.cpp on Dankclaw;
|
||||||
llama.cpp FastMTP on Dankserver.
|
`frank-default-small` runs the upstream llama.cpp on Dankserver.
|
||||||
|
|
||||||
## Boot Persistence
|
## Boot Persistence
|
||||||
|
|
||||||
@@ -58,4 +58,4 @@ Deploy the SearXNG trio with `scripts/deploy-searxng-quadlet.sh dankclaw` and op
|
|||||||
- [[index]] — repository index
|
- [[index]] — repository index
|
||||||
- [[Network Architecture]] — routing and proxy
|
- [[Network Architecture]] — routing and proxy
|
||||||
- [[Observability]] — Grafana, Prometheus, logs
|
- [[Observability]] — Grafana, Prometheus, logs
|
||||||
- [vllmScripts knowledge base](../../vllmScripts/docs/index.md) — the model-serving endpoints on this host
|
- [LLM fleet knowledge base](../LLM/index.md) — the model-serving endpoints on this host
|
||||||
@@ -18,10 +18,10 @@ The containers share the private bridge network `systemd-gitea`. The quadlet net
|
|||||||
|
|
||||||
| Repository | Purpose |
|
| Repository | Purpose |
|
||||||
|------------|---------|
|
|------------|---------|
|
||||||
| `dank/serverAppConfigs` | this repo — Podman stack configs |
|
| `dank/serverAppConfigs` | this repo — Podman stacks + LLM fleet |
|
||||||
| `dank/vllmScripts` | model-serving scripts: llama.cpp FastMTP on Dankserver (`:7770`), vLLM on Dankclaw (`:52625`) |
|
| `dank/vllmScripts` | retired 2026-08-22 — merged into `dank/serverAppConfigs` (history preserved) |
|
||||||
|
|
||||||
The vllmScripts knowledge base: [../vllmScripts/docs/index.md](../../vllmScripts/docs/index.md)
|
The LLM fleet knowledge base: [docs/LLM/index.md](../LLM/index.md)
|
||||||
|
|
||||||
## Management
|
## Management
|
||||||
|
|
||||||
@@ -83,7 +83,7 @@ Back up `/opt/gitea/data/` (tar) or `pg_dump` the database. See the knowledge ba
|
|||||||
- **The files in the data directory are owned by host UID 100999**, not `dank` (rootless ID mapping: container uid 1000 → host 100999). Edit them inside the container with `podman exec --user 1000 gitea-server ...`.
|
- **The files in the data directory are owned by host UID 100999**, not `dank` (rootless ID mapping: container uid 1000 → host 100999). Edit them inside the container with `podman exec --user 1000 gitea-server ...`.
|
||||||
- **The host auto-reboots.** `unattended-upgrades` reboots Dankclaw when a package needs it. The quadlet units make Gitea survive that. See the 2026-08-16 incident below.
|
- **The host auto-reboots.** `unattended-upgrades` reboots Dankclaw when a package needs it. The quadlet units make Gitea survive that. See the 2026-08-16 incident below.
|
||||||
- **Port 3000 vs 3001.** The live host port is 3000. The full-stack [`dankclaw/compose.yaml`](../compose.yaml) maps 3001. Align the two before deploying the full stack.
|
- **Port 3000 vs 3001.** The live host port is 3000. The full-stack [`dankclaw/compose.yaml`](../compose.yaml) maps 3001. Align the two before deploying the full stack.
|
||||||
- **Legacy compose retired.** `/opt/gitea/podman-compose.yml` is retired to `/opt/gitea/.deprecated/`. Do not run it; its containers would collide with the quadlet units on ports 3000, 2222, 80, and 443.
|
- **Legacy compose retired.** `/opt/gitea/podman-compose.yml` is retired to `/opt/gitea/.deprecated/`. Do not run it. Its containers collide with the quadlet units on ports 3000, 2222, 80, and 443.
|
||||||
- **Rootless cannot bind ports below 1024 by default.** `gitea-caddy` publishes 80 and 443. Without a host sysctl, `rootlessport` fails with `permission denied` and the unit crash-loops under `Restart=always`. The fix is a one-time sysctl: `net.ipv4.ip_unprivileged_port_start=80`, persisted in `/etc/sysctl.d/99-rootless-ports.conf`. Applied 2026-08-18 after a 13,000-restart crash loop.
|
- **Rootless cannot bind ports below 1024 by default.** `gitea-caddy` publishes 80 and 443. Without a host sysctl, `rootlessport` fails with `permission denied` and the unit crash-loops under `Restart=always`. The fix is a one-time sysctl: `net.ipv4.ip_unprivileged_port_start=80`, persisted in `/etc/sysctl.d/99-rootless-ports.conf`. Applied 2026-08-18 after a 13,000-restart crash loop.
|
||||||
|
|
||||||
## Incident: 2026-08-16 — down after unattended reboot
|
## Incident: 2026-08-16 — down after unattended reboot
|
||||||
|
|||||||
+73
-35
@@ -1,10 +1,19 @@
|
|||||||
# Deployment and Update Process
|
# Deployment and Update Process
|
||||||
|
|
||||||
This page describes how to deploy and update the LLM servers on Dankserver and Dankclaw. The process uses git as the single source of truth. All script edits flow through this repository.
|
This page describes how to deploy and update the LLM servers on Dankserver
|
||||||
|
and Dankclaw. The process uses git as the single source of truth. All script
|
||||||
|
edits flow through this repository.
|
||||||
|
|
||||||
|
> 2026-08-22: the former `vllmScripts` repository is merged into this repo.
|
||||||
|
> Both LLM services now run from the `/home/dank/serverAppConfigs` checkout
|
||||||
|
> on each host. The retired checkouts (`/home/dank/vLLM` on Dankserver,
|
||||||
|
> `/home/dank/vllmScripts` on Dankclaw) are archived on the hosts.
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
Two servers run LLM inference services. Each server pulls its scripts from this repository. The deployment process moves a committed change from the repository to a running service.
|
Two servers run LLM inference services. Each server pulls its scripts from
|
||||||
|
this repository. The deployment process moves a committed change from the
|
||||||
|
repository to a running service.
|
||||||
|
|
||||||
| Target | Host | Runtime | Service | Port |
|
| Target | Host | Runtime | Service | Port |
|
||||||
|--------|------|---------|---------|------|
|
|--------|------|---------|---------|------|
|
||||||
@@ -13,38 +22,42 @@ Two servers run LLM inference services. Each server pulls its scripts from this
|
|||||||
|
|
||||||
## Workflow
|
## Workflow
|
||||||
|
|
||||||
The deployment workflow has six steps.
|
The deployment workflow has seven steps.
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
graph LR
|
graph LR
|
||||||
A[Edit scripts locally] --> B[Commit]
|
A[Edit scripts locally] --> B[Commit]
|
||||||
B --> C[Push to origin<br>Gitea on Dankclaw]
|
B --> C[Push to origin<br>Gitea on Dankclaw]
|
||||||
C --> D[Pull on target<br>git pull]
|
C --> D[Pull on target<br>git pull]
|
||||||
D --> E[Restart systemd service]
|
D --> E[Restart podman stack]
|
||||||
E --> F[Verify health endpoint]
|
E --> F[Relink + restart LLM unit]
|
||||||
|
F --> G[Verify health endpoints]
|
||||||
```
|
```
|
||||||
|
|
||||||
1. Fetch the current script from the target machine.
|
1. Fetch the current script from the target machine.
|
||||||
2. Commit the fetched version in this repository.
|
2. Commit the fetched version in this repository.
|
||||||
3. Edit the script locally in the working tree.
|
3. Edit the script locally in the working tree.
|
||||||
4. Commit and push the change to origin.
|
4. Run the tests (`./tests/run.sh` must pass).
|
||||||
5. Pull the change on the target machine.
|
5. Commit and push the change to origin.
|
||||||
6. Restart the service and verify that it is healthy.
|
6. Pull the change on the target machine.
|
||||||
|
7. Restart the services and verify that they are healthy.
|
||||||
|
|
||||||
## Deploy Script
|
## Deploy Script
|
||||||
|
|
||||||
The `scripts/deploy.sh` script executes steps 4 through 6. Run it from the repository root.
|
The `scripts/deploy.sh` script is the unified deployer. One run per target
|
||||||
|
covers both workloads on that host: the podman stack and the LLM service.
|
||||||
|
Run it from the repository root.
|
||||||
|
|
||||||
Run this command to deploy to Dankserver:
|
Run this command to deploy to Dankserver:
|
||||||
|
|
||||||
```
|
```
|
||||||
FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver
|
./scripts/deploy.sh dankserver
|
||||||
```
|
```
|
||||||
|
|
||||||
Run this command to deploy to Dankclaw:
|
Run this command to deploy to Dankclaw:
|
||||||
|
|
||||||
```
|
```
|
||||||
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw
|
./scripts/deploy.sh dankclaw
|
||||||
```
|
```
|
||||||
|
|
||||||
Add the `--dry-run` flag to preview without making changes:
|
Add the `--dry-run` flag to preview without making changes:
|
||||||
@@ -58,12 +71,18 @@ The script does these actions:
|
|||||||
1. Verifies that the local working tree has no uncommitted changes.
|
1. Verifies that the local working tree has no uncommitted changes.
|
||||||
2. Pushes the `main` branch to origin.
|
2. Pushes the `main` branch to origin.
|
||||||
3. Pulls the repository on the target machine.
|
3. Pulls the repository on the target machine.
|
||||||
4. Symlinks the systemd unit, reloads systemd, enables, and restarts the service.
|
4. Restarts the podman stack (Dankclaw only. The Dankserver media stack
|
||||||
5. Waits for the health endpoint to respond (over ssh).
|
runs as bare containers).
|
||||||
|
5. Relinks the LLM systemd units, reloads systemd, enables, and restarts
|
||||||
|
the active LLM service.
|
||||||
|
6. Waits for the Caddy homepage and the LLM health endpoint (over ssh).
|
||||||
|
|
||||||
Set `FRANK_SUDO_PW` when running headless, so that remote sudo reads the
|
Set `FRANK_SUDO_PW` when running headless on a host without passwordless
|
||||||
password from stdin (`sudo -S`). The passwordless restart helpers from the
|
sudo, so that remote sudo reads the password from stdin (`sudo -S`). The
|
||||||
retired topology are obsolete and are not used.
|
password is never stored in this repo. Both fleet hosts currently allow
|
||||||
|
passwordless sudo for the deploy user (verified 2026-08-22). The
|
||||||
|
passwordless restart helpers from the retired topology are obsolete and are
|
||||||
|
not used.
|
||||||
|
|
||||||
Health endpoints:
|
Health endpoints:
|
||||||
|
|
||||||
@@ -77,9 +96,9 @@ Health endpoints:
|
|||||||
Run these commands on Dankserver:
|
Run these commands on Dankserver:
|
||||||
|
|
||||||
```
|
```
|
||||||
cd /home/dank/vLLM
|
cd /home/dank/serverAppConfigs
|
||||||
git pull origin main
|
git pull origin main
|
||||||
sudo ln -sf /home/dank/vLLM/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
|
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
|
||||||
sudo systemctl daemon-reload
|
sudo systemctl daemon-reload
|
||||||
sudo systemctl restart frank-default-small
|
sudo systemctl restart frank-default-small
|
||||||
curl -sf http://127.0.0.1:7770/health
|
curl -sf http://127.0.0.1:7770/health
|
||||||
@@ -90,9 +109,9 @@ curl -sf http://127.0.0.1:7770/health
|
|||||||
Run these commands on Dankclaw:
|
Run these commands on Dankclaw:
|
||||||
|
|
||||||
```
|
```
|
||||||
cd /home/dank/vllmScripts
|
cd /home/dank/serverAppConfigs
|
||||||
git pull origin main
|
git pull origin main
|
||||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
|
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
|
||||||
sudo systemctl daemon-reload
|
sudo systemctl daemon-reload
|
||||||
sudo systemctl restart frank-default-large
|
sudo systemctl restart frank-default-large
|
||||||
curl -sf http://127.0.0.1:52625/health
|
curl -sf http://127.0.0.1:52625/health
|
||||||
@@ -117,11 +136,12 @@ launches through `scripts/serve-frank-small.sh`, which reads tunables from
|
|||||||
|
|
||||||
The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
|
The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
|
||||||
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v3 five-file payload: GGUF
|
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v3 five-file payload: GGUF
|
||||||
+ FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj) under the alias `frank-default-large` on port 52625. It runs
|
+ FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj) under the alias
|
||||||
|
`frank-default-large` on port 52625. It runs
|
||||||
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
|
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
|
||||||
7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP
|
7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP
|
||||||
(`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context
|
(`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context
|
||||||
(1 slot; concurrent requests queue, the server never preempts a running
|
(1 slot. Concurrent requests queue. The server never preempts a running
|
||||||
generation), batch 4096 / ubatch 4096, 16 context checkpoints, 8 GiB
|
generation), batch 4096 / ubatch 4096, 16 context checkpoints, 8 GiB
|
||||||
prompt cache, and reasoning in deepseek format. The unit file is
|
prompt cache, and reasoning in deepseek format. The unit file is
|
||||||
`scripts/frank-default-large.service`. It launches through
|
`scripts/frank-default-large.service`. It launches through
|
||||||
@@ -129,11 +149,12 @@ prompt cache, and reasoning in deepseek format. The unit file is
|
|||||||
`scripts/frank-default-large.env`. See [[ciru-activefpx-promptforge]].
|
`scripts/frank-default-large.env`. See [[ciru-activefpx-promptforge]].
|
||||||
|
|
||||||
> Restored 2026-08-21. The intermediate DFlash2 Vulkan deployment was
|
> Restored 2026-08-21. The intermediate DFlash2 Vulkan deployment was
|
||||||
> retired the same day; see [[dflash2-large-vulkan]] for the archived runbook.
|
> retired the same day. See [[dflash2-large-vulkan]] for the archived runbook.
|
||||||
|
|
||||||
## Validation
|
## Validation
|
||||||
|
|
||||||
The deploy script checks the health endpoint after a restart. For deeper checks, run a completion request against the endpoint:
|
The deploy script checks the health endpoint after a restart. For deeper
|
||||||
|
checks, run a completion request against the endpoint:
|
||||||
|
|
||||||
```
|
```
|
||||||
curl -s http://127.0.0.1:7770/v1/chat/completions \
|
curl -s http://127.0.0.1:7770/v1/chat/completions \
|
||||||
@@ -152,19 +173,36 @@ If the health check fails after a deploy:
|
|||||||
|
|
||||||
## Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13)
|
## Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13)
|
||||||
|
|
||||||
On 2026-08-13 the old `frank-default-large` service crash-looped for hours on Dankclaw. Root cause: the env files exported `TEMP=1.0` (a sampling default). The ROCm HIP runtime reads `TEMP` as its JIT kernel-cache directory, used the path `1.0`, failed, and segfaulted on a NULL object. Fix: rename the sampling variable to `TEMPERATURE` (commit `227ce01`).
|
On 2026-08-13 the old `frank-default-large` service crash-looped for hours
|
||||||
|
on Dankclaw. Root cause: the env files exported `TEMP=1.0` (a sampling
|
||||||
|
default). The ROCm HIP runtime reads `TEMP` as its JIT kernel-cache
|
||||||
|
directory, used the path `1.0`, failed, and segfaulted on a NULL object.
|
||||||
|
Fix: rename the sampling variable to `TEMPERATURE` (commit `227ce01`).
|
||||||
|
|
||||||
If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Do not name any tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`.
|
If a service segfaults at load with no obvious cause, compare the exported
|
||||||
|
env against the HIP runtime's env reads. Do not name any tunable `TEMP`,
|
||||||
|
`HOST`, or `GPU_MAX_HEAP_SIZE`.
|
||||||
|
|
||||||
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
|
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
|
||||||
|
|
||||||
Historical: on the retired topology, parallel jobs on the Dankserver 35B service died with a cuBLAS crash under researcher load. The affected service is retired. A client abort logs `W srv stop: cancel task, id_task = N`; a natural stop logs only `slot release` with `truncated = 0`. The server never preempts a running generation.
|
Historical: on the retired topology, parallel jobs on the Dankserver 35B
|
||||||
|
service died with a cuBLAS crash under researcher load. The affected
|
||||||
|
service is retired. A client abort logs `W srv stop: cancel task,
|
||||||
|
id_task = N`. A natural stop logs only `slot release` with
|
||||||
|
`truncated = 0`. The server never preempts a running generation.
|
||||||
|
|
||||||
## Incident: frank-default-small Crash Loop After Distro Upgrade (2026-08-21)
|
## Incident: frank-default-small Crash Loop After Distro Upgrade (2026-08-21)
|
||||||
|
|
||||||
On 2026-08-21 a release upgrade moved Dankserver from Ubuntu 24.04 to Ubuntu 26.04 (Resolute Raccoon). The upgrade removed the CUDA 12 runtime packages: `libcudart12`, `libcublas12`, `libcublaslt12`, and `nvidia-cuda-toolkit`. The `llama-server` binary links `libcudart.so.12` and `libcublas.so.12`. On every start the binary exited with code 127. systemd restart-looped the service continuously. The health endpoint on port 7770 stayed down.
|
On 2026-08-21 a release upgrade moved Dankserver from Ubuntu 24.04 to
|
||||||
|
Ubuntu 26.04 (Resolute Raccoon). The upgrade removed the CUDA 12 runtime
|
||||||
|
packages: `libcudart12`, `libcublas12`, `libcublaslt12`, and
|
||||||
|
`nvidia-cuda-toolkit`. The `llama-server` binary links
|
||||||
|
`libcudart.so.12` and `libcublas.so.12`. On every start the binary exited
|
||||||
|
with code 127. systemd restart-looped the service continuously. The health
|
||||||
|
endpoint on port 7770 stayed down.
|
||||||
|
|
||||||
Fix: reinstall the runtime packages. The same versions remain in the 26.04 repository.
|
Fix: reinstall the runtime packages. The same versions remain in the 26.04
|
||||||
|
repository.
|
||||||
|
|
||||||
sudo apt-get install -y libcudart12 libcublas12 libcublaslt12
|
sudo apt-get install -y libcudart12 libcublas12 libcublaslt12
|
||||||
|
|
||||||
@@ -174,13 +212,13 @@ Verify that the binary resolves all libraries:
|
|||||||
|
|
||||||
Start the service and check the health endpoint on port 7770.
|
Start the service and check the health endpoint on port 7770.
|
||||||
|
|
||||||
After any release upgrade on Dankserver, run the `ldd` check above. A missing CUDA 12 library means the packages need a reinstall.
|
After any release upgrade on Dankserver, run the `ldd` check above. A
|
||||||
|
missing CUDA 12 library means the packages need a reinstall.
|
||||||
|
|
||||||
## See Also
|
## See Also
|
||||||
|
|
||||||
- [AGENTS.md](../AGENTS.md) — the git workflow rules
|
- [AGENTS.md](../../AGENTS.md) — the git workflow rules
|
||||||
- [README.md](../README.md) — repository overview
|
- [README.md](../../README.md) — repository overview
|
||||||
- [index.md](index.md) — knowledge base index
|
- [index.md](index.md) — LLM fleet knowledge base index
|
||||||
- [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) — the full redo runbook
|
- [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) — the full redo runbook
|
||||||
- [serverAppConfigs knowledge base](../serverAppConfigs/docs/index.md) —
|
- [docs/index.md](../index.md) — the project knowledge base (podman stacks)
|
||||||
the sibling Podman app stack on the same hosts
|
|
||||||
+16
-9
@@ -1,10 +1,15 @@
|
|||||||
# vLLM Scripts — Knowledge Base
|
# LLM Fleet — Knowledge Base
|
||||||
|
|
||||||
Documentation for the LLM inference fleet on two local machines
|
Documentation for the LLM inference fleet on two local machines
|
||||||
(2026-08-22): upstream llama.cpp on Dankserver (frank-default-small) and
|
(2026-08-22): upstream llama.cpp on Dankserver (frank-default-small) and
|
||||||
CIRU ROCmFPX llama.cpp with the PromptForge v3 runtime on Dankclaw
|
CIRU ROCmFPX llama.cpp with the PromptForge v3 runtime on Dankclaw
|
||||||
(frank-default-large).
|
(frank-default-large).
|
||||||
|
|
||||||
|
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||||
|
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||||
|
> `docs/` root to `docs/LLM/`. The fleet scripts and units now live in the
|
||||||
|
> repo's `scripts/` directory.
|
||||||
|
|
||||||
## Deployment Topology
|
## Deployment Topology
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
@@ -43,11 +48,11 @@ frank-default-large, port 52625]
|
|||||||
- [[deployment]] — deploy and update process for both servers
|
- [[deployment]] — deploy and update process for both servers
|
||||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||||
- [[ciru-activefpx-promptforge]] — the v3 CIRU PromptForge runtime on
|
- [[ciru-activefpx-promptforge]] — the v3 CIRU PromptForge runtime on
|
||||||
Dankclaw (frank-default-large, active; restored 2026-08-22)
|
Dankclaw (frank-default-large, active. Restored 2026-08-22)
|
||||||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
||||||
(deployed and reverted 2026-08-22)
|
(deployed and reverted 2026-08-22)
|
||||||
- [[Dankserver]] — the upstream llama.cpp box (frank-default-small)
|
- [[Dankserver System Specs]] — the upstream llama.cpp box (frank-default-small)
|
||||||
- [[Dankclaw]] — the CIRU ROCmFPX box (frank-default-large)
|
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box (frank-default-large)
|
||||||
|
|
||||||
## CIRU Reference (Dankclaw)
|
## CIRU Reference (Dankclaw)
|
||||||
|
|
||||||
@@ -88,10 +93,12 @@ frank-default-large, port 52625]
|
|||||||
## Pipeline Parallelism (archive)
|
## Pipeline Parallelism (archive)
|
||||||
|
|
||||||
- [[Pipeline Parallelism]] — index of multi-node inference research. The PP
|
- [[Pipeline Parallelism]] — index of multi-node inference research. The PP
|
||||||
pair is retired; these pages are research history.
|
pair is retired. These pages are research history.
|
||||||
|
|
||||||
## Sibling projects
|
## Related
|
||||||
|
|
||||||
- [serverAppConfigs](../serverAppConfigs/docs/index.md) — the Podman app
|
- [Project knowledge base](../index.md) — the Podman app stacks on the same
|
||||||
stack on the same two hosts. Its OpenWebUI consumes the `:7770` endpoint
|
two hosts. OpenWebUI consumes the `:7770` endpoint on Dankserver. The
|
||||||
on Dankserver; its Gitea instance hosts this repository.
|
Gitea instance on Dankclaw hosts this repository.
|
||||||
|
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the unified
|
||||||
|
git-based deploy for both workloads
|
||||||
+66
-5
@@ -2,7 +2,14 @@
|
|||||||
|
|
||||||
Podman deployment configuration for two homelab servers: Dankserver and Dankclaw.
|
Podman deployment configuration for two homelab servers: Dankserver and Dankclaw.
|
||||||
|
|
||||||
This repository tracks the container stacks for both hosts. Git is the single source of truth. Edits flow through this repo, then deploy to the targets with `scripts/deploy.sh`. This mirrors the workflow of the `vllmScripts` repository.
|
This repository is the single source of truth for both hosts. It tracks the
|
||||||
|
container stacks and the LLM inference fleet. Edits flow through this repo,
|
||||||
|
then deploy to the targets with `scripts/deploy.sh` (one script per target
|
||||||
|
covers both workloads).
|
||||||
|
|
||||||
|
> 2026-08-22: the former `vllmScripts` repository (LLM fleet) is merged
|
||||||
|
> into this repo with its full git history. Its knowledge base now lives at
|
||||||
|
> [docs/LLM/index.md](LLM/index.md).
|
||||||
|
|
||||||
## Hosts
|
## Hosts
|
||||||
|
|
||||||
@@ -74,10 +81,64 @@ This repository tracks the container stacks for both hosts. Git is the single so
|
|||||||
- [[Jellyfin]] — media server research brief
|
- [[Jellyfin]] — media server research brief
|
||||||
- [[ARR-Stack]] — automated download stack research brief
|
- [[ARR-Stack]] — automated download stack research brief
|
||||||
|
|
||||||
|
## LLM Fleet
|
||||||
|
|
||||||
|
The LLM inference fleet on the two hosts. Full index:
|
||||||
|
[docs/LLM/index.md](LLM/index.md).
|
||||||
|
|
||||||
|
### Deployment and Topology
|
||||||
|
|
||||||
|
- [[deployment]] — unified deploy and update process for both servers
|
||||||
|
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||||
|
- [[ciru-activefpx-promptforge]] — the v3 CIRU PromptForge runtime on Dankclaw (active)
|
||||||
|
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22)
|
||||||
|
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21)
|
||||||
|
- [[Dankserver System Specs]] — the upstream llama.cpp box hardware
|
||||||
|
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box hardware
|
||||||
|
- [[Dankclaw Improvements]] — host-level changes on the CIRU box
|
||||||
|
|
||||||
|
### Benchmarks and Sizing
|
||||||
|
|
||||||
|
- [[frank-default-large-benchmark]] — backend/spec benchmark behind the old head
|
||||||
|
- [[frank-default-small-benchmark]] — the small-model benchmark suite
|
||||||
|
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware notes
|
||||||
|
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Qwen3.6-27B sizing on the iGPU
|
||||||
|
- [[qwen3.8-strix-optimizations]] — Strix Halo tuning notes
|
||||||
|
|
||||||
|
### Memory and Hardware
|
||||||
|
|
||||||
|
- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
|
||||||
|
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
|
||||||
|
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
|
||||||
|
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
|
||||||
|
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
|
||||||
|
|
||||||
|
### Retired Topology and History
|
||||||
|
|
||||||
|
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
|
||||||
|
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
|
||||||
|
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
|
||||||
|
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
|
||||||
|
- [[ling-3.0-flash-trial]] — the Ling-3.0-Flash trial runbook
|
||||||
|
- [[ling-3.0-flash-vllm-plan]] — the Ling-3.0-Flash vLLM plan (retired)
|
||||||
|
|
||||||
|
### vLLM Reference (historical)
|
||||||
|
|
||||||
|
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
|
||||||
|
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
|
||||||
|
- [[vllm_research_findings]] — research findings from the vLLM era
|
||||||
|
|
||||||
|
### Pipeline Parallelism (archive)
|
||||||
|
|
||||||
|
- [[Ethernet Bandwidth Math]] — the bandwidth budget for the PP pair
|
||||||
|
- [[Expected Performance]] — expected PP throughput
|
||||||
|
- [[llama.cpp RPC]] — the RPC transport between head and worker
|
||||||
|
- [[Mixed-Vendor Experiments]] — cross-vendor PP experiments
|
||||||
|
- [[Qwen3.6 27B Spec Methods]] — spec-decoding methods for the 27B
|
||||||
|
- [[Sample Script Set]] — reference scripts for the PP pair
|
||||||
|
- [[vLLM Pipeline Parallelism]] — vLLM's PP implementation
|
||||||
|
|
||||||
## See Also
|
## See Also
|
||||||
|
|
||||||
- [vllmScripts repository](../vllmScripts/docs/index.md) — the model-serving
|
|
||||||
scripts on the two hosts (2026-08-17 redo): `frank-default-large`
|
|
||||||
(llama.cpp FastMTP, Dankserver, port 7770) and `frank-default-small`
|
|
||||||
(vLLM, Dankclaw, port 52625). Git workflow model for this repo.
|
|
||||||
- [[AGENTS.md]] — repository rules
|
- [[AGENTS.md]] — repository rules
|
||||||
|
- [docs/LLM/index.md](LLM/index.md) — the LLM fleet knowledge base
|
||||||
Reference in new issue
Block a user