dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Deploy services independently by changed paths

Browse Source
This commit is contained in:
Frank
2026-08-25 20:13:33 -07:00
parent 2277a2e9ac
commit 1ba47e4d22
18 changed files with 1462 additions and 894 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Architecture/Deployment Flow.md
+99 -30
View File
@@ -1,60 +1,129 @@
---
type: guide
title: Deployment Flow
description: Service-scoped Git deployment for the two homelab hosts.
tags:
- deployment
- operations
- git
timestamp: 2026-09-03
---
# Deployment Flow
Git is the single source of truth. A `deploy.sh` script moves committed
changes to the target hosts over SSH. One run per target covers both
workloads on that host: the podman stack and the LLM service.
Git is the single source of truth. `scripts/deploy.sh` pushes the committed
`main` branch, pulls it on one target, and applies only the runtime services
whose tracked inputs changed. A SearXNG change no longer recreates the
inference service; an LLM change no longer restarts unrelated containers.
## Workflow
```mermaid
graph LR
A[Edit in this repo] --> B[Commit]
B --> C[Push to origin]
C --> D[deploy.sh: pull on target]
D --> E[restart stack]
E --> F[health check]
A[Edit in this repo] --> B[Run tests]
B --> C[Commit main]
C --> D[deploy.sh computes plan]
D --> E[Push main]
E --> F[Target pulls with ff-only]
F --> G[deploy-remote.sh dispatches selected services]
G --> H[Per-service health check]
```
1. Edit files locally in this repository.
2. Commit and push.
3. Run `./scripts/deploy.sh <dankclaw|dankserver>`.
4. The script pulls the repo on the target and restarts the podman stack
and the LLM service.
5. It verifies service health after restart (homepage and LLM endpoint).
2. Run `./tests/run.sh`, then commit the complete change on `main`.
3. Run `./scripts/deploy.sh <target>`.
4. The planner maps changed paths to services. Compose YAML is compared by
service definition so an edit to one service does not restart its peers.
5. The target pulls with `git pull --ff-only origin main` and runs the remote
dispatcher with the explicit plan.
6. The dispatcher updates only selected Quadlet, compose, homepage, Caddy, or
systemd services and records each successful service revision outside Git.
## Deploy Script
## Commands
```
./scripts/deploy.sh dankclaw # pull + restart on Dankclaw
./scripts/deploy.sh dankserver # pull + restart on Dankserver
```bash
./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankserver
./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-default-small
./scripts/deploy.sh dankclaw --dry-run
```
SSH uses the existing key pair (`dank@dankclaw`, `dank@dankserver`). No passwords are stored or documented in this repo. If sudo is needed, use the interactive sudo on the target — never commit a password.
`--service` is repeatable and is an explicit retry/override. Without it, the
planner compares the local `HEAD` to `origin/main`. A documentation-only
commit produces an empty plan. The remote state directory can also replay a
service whose earlier deployment failed, but it never expands a plan to a
service on the other host.
## Service managers
| Service family | Runtime manager | Scope behavior |
|---|---|---|
| Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
| SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. |
| `frank-default-large` | systemd + `systemctl` | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
| `frank-default-small` | systemd + `systemctl` | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
| Dankserver media inventory | host-managed bare containers | Compose is inventory only; the scoped deploy does not create duplicate containers. |
The shared chat template maps to the inference service on the selected target.
The inactive `frank-default-npu` unit is relink-only when explicitly selected;
it is not restarted by an unrelated large-model deployment.
## Caddy safety
Caddy runs on Dankclaw and fronts both hosts. The dispatcher never stops
`gitea-caddy` implicitly. If the legacy `gitea-caddy` container owns ports
80/443 while the compose `caddy` container is absent or stopped, a Caddy
selection fails closed rather than replacing the live proxy. Resolve port
ownership explicitly before retrying. A Caddy reload failure falls back to a
restart of **only** the selected `caddy` container.
Homepage files, generated `deployed.js`, the status checker, and the rendered
Caddyfile are updated only when their service is selected. A media-only
Dankserver deployment does not touch Dankclaw's proxy or homepage.
## Per-service state
The remote runner writes one revision file per successfully completed service
to:
```text
${XDG_STATE_HOME:-~/.local/state}/serverAppConfigs/deploy-state/
```
The directory is outside the checkout and is never committed. A failed handler
is not marked successful. On the next ordinary deployment, the runner compares
the recorded revision with the new checkout and retries only a still-changed
service. This prevents a failed Caddy or LLM deployment from being silently
forgotten while avoiding a host-wide restart.
## Targets
| Target | Host | Stack |
|--------|------|-------|
| dankclaw | dank@dankclaw (192.168.0.143) | `dankclaw/` |
| dankserver | dank@dankserver (192.168.0.235) | `dankserver/` |
| Target | Host | Repository checkout |
|---|---|---|
| dankclaw | `dank@dankclaw` (`192.168.0.143`) | `/home/dank/serverAppConfigs` |
| dankserver | `dank@dankserver` (`192.168.0.235`) | `/home/dank/serverAppConfigs` |
## Health Verification
The deploy script polls the health endpoint of each service after restart. Service healthchecks are defined in each compose file. A service that fails its healthcheck is reported, not hidden.
SSH uses the existing key pair. No password is stored or documented in this
repository. If sudo is needed on a headless target, `FRANK_SUDO_PW` is read
from the environment and is never written to the repository.
## Tests
Run the test suite before pushing:
Run the full suite before pushing:
```
```bash
./tests/run.sh
```
The suite validates compose YAML, env consistency, Caddyfiles, docs completeness, and script syntax. See [[Testing Policy]].
The deployment seams are in `tests/deployment_scoping.bats` and
`tests/deployment_flow.bats`. They cover path isolation, compose narrowing,
Caddy safety, inference isolation, per-service state, and legacy wrapper
routing. See [[Testing Policy]].
## See Also
- [[Storage Policy]]
- [[Testing Policy]]
- [LLM deployment docs](../LLM/deployment.md) — the LLM half of the same flow
- [LLM deployment](../LLM/deployment.md)
- [[Caddy]]
- [[SearXNG]]
docs/Architecture/Testing Policy.md
+1 -1
View File
@@ -11,7 +11,7 @@ This repository tests its configuration at defined seams. The tests run with `ba
| Caddyfile validity | `tests/caddyfile.bats` | per-host Caddyfile, `.local` convention, balanced braces, basic auth present |
| Docs completeness | `tests/docs_completeness.bats` | every compose service has a docs page; index links every page |
| Script syntax | `tests/scripts_syntax.bats` | `bash -n` clean, executable bits, shellcheck when available |
| Deployment flow | `tests/deployment_flow.bats` | deploy.sh exists, targets known, SSH reachable (skips offline) |
| Deployment flow | `tests/deployment_flow.bats`, `tests/deployment_scoping.bats` | scoped dispatcher exists, targets are known, service plans isolate inference, Caddy ownership is guarded, and SSH reachability is checked (skips offline) |
## Focused pre-commit testing
docs/LLM/deployment.md
+96 -199
View File
@@ -1,227 +1,124 @@
---
type: guide
title: LLM Deployment
description: Scoped deployment and update process for the two inference services.
tags:
- llm
- deployment
- systemd
timestamp: 2026-09-03
---
# Deployment and Update Process
This page describes how to deploy and update the LLM servers on Dankserver
and Dankclaw. The process uses git as the single source of truth. All script
edits flow through this repository.
This page describes how to deploy and update the LLM servers on Dankserver and
Dankclaw. The process uses Git as the single source of truth. A change to
SearXNG, Caddy, or another container service does **not** restart inference.
> 2026-08-22: the former `vllmScripts` repository is merged into this repo.
> Both LLM services now run from the `/home/dank/serverAppConfigs` checkout
> on each host. The retired checkouts (`/home/dank/vLLM` on Dankserver,
> `/home/dank/vllmScripts` on Dankclaw) are archived on the hosts.
## Fleet
## Overview
| Target | Host | Service | Port |
|---|---|---|---:|
| Dankserver | `dank@dankserver` | `frank-default-small` | 7770 |
| Dankclaw | `dank@dankclaw` | `frank-default-large` | 7770 |
Two servers run LLM inference services. Each server pulls its scripts from
this repository. The deployment process moves a committed change from the
repository to a running service.
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankserver | dank@dankserver | upstream llama.cpp (Ornith-1.5-35B-A3B-CRACK Q6_K, native MTP draft, selected MoE experts offloaded via --override-tensor) | frank-default-small | 7770 |
| Dankclaw | dank@dankclaw | charlie12345/ROCmFPX llama.cpp (kingjones Qwen3.8 Uncensored Q6_0 ROCmFPX AGENT, standalone MTP, 262K ctx / 1 slot) | frank-default-large | 7770 |
## Workflow
The deployment workflow has seven steps.
## Scoped workflow
```mermaid
graph LR
A[Edit scripts locally] --> B[Commit]
B --> C[Push to origin<br>Gitea on Dankclaw]
C --> D[Pull on target<br>git pull]
D --> E[Restart podman stack]
E --> F[Relink + restart LLM unit]
F --> G[Verify health endpoints]
A[Edit LLM files] --> B[Run tests]
B --> C[Commit main]
C --> D[deploy.sh selects matching service]
D --> E[Target pulls ff-only]
E --> F[Relink one unit]
F --> G[Restart one inference service]
G --> H[Wait for that service health]
```
1. Fetch the current script from the target machine.
2. Commit the fetched version in this repository.
3. Edit the script locally in the working tree.
4. Run the tests (`./tests/run.sh` must pass).
5. Commit and push the change to origin.
6. Pull the change on the target machine.
7. Restart the services and verify that they are healthy.
Run the unified deployer from the repository root:
## Deploy Script
The `scripts/deploy.sh` script is the unified deployer. One run per target
covers both workloads on that host: the podman stack and the LLM service.
Run it from the repository root.
Run this command to deploy to Dankserver:
```
./scripts/deploy.sh dankserver
```bash
./scripts/deploy.sh dankserver --service frank-default-small
./scripts/deploy.sh dankclaw --service frank-default-large
./scripts/deploy.sh dankclaw --dry-run
```
Run this command to deploy to Dankclaw:
Without `--service`, `scripts/deploy-plan.py` maps changed files to the service
on the selected host. The following files select the corresponding service:
```
./scripts/deploy.sh dankclaw
- `scripts/frank-default-small.env`, `frank-default-small.service`, and
`serve-frank-small.sh` select `frank-default-small` on Dankserver.
- `scripts/frank-default-large.env`, `frank-default-large.service`, and
`serve-frank-large.sh` select `frank-default-large` on Dankclaw.
- `scripts/chat-template.jinja` selects the LLM on the selected host only.
- Documentation, tests, deployment tooling, and unrelated container changes
do not select either inference service.
The target-side `scripts/deploy-remote.sh` relinks the selected unit into
`/etc/systemd/system/`, reloads systemd, enables it, restarts it, and waits for
`http://127.0.0.1:7770/health`. The wait budget is 20 minutes because cold model
loads can be slow. A failed handler does not advance its per-service state.
## Unit and artifact paths
Both units point into the stable target checkout:
```text
/home/dank/serverAppConfigs/scripts/frank-default-small.service
/home/dank/serverAppConfigs/scripts/frank-default-large.service
```
Add the `--dry-run` flag to preview without making changes:
The unit files and serve scripts validate their model/runtime artifacts before
starting. Keep `REMOTE_DIR` stable; moving the checkout requires updating both
the unit absolute paths and the deployment tests in the same change.
```
./scripts/deploy.sh dankserver --dry-run
## NPU unit
`frank-default-npu` is an inactive, historical Dankclaw endpoint. It is
relinked only when explicitly selected with `--service frank-default-npu`; the
large-model handler never restarts it. This is intentional: the NPU is not part
of the active inference path and should not be disturbed by a large-model
change.
## Isolation guarantee
The remote runner has one explicit service dispatch. It does not run a
host-wide LLM loop, `podman-compose up` for unrelated services, or a blanket
systemd restart. Each successful service is recorded under the target user's
state directory outside Git:
```text
~/.local/state/serverAppConfigs/deploy-state/
```
The script does these actions:
On a later ordinary deployment, a service remains pending only while its
recorded revision differs from the checkout and the planner still finds a
change in that service's inputs.
1. Verifies that the local working tree has no uncommitted changes.
2. Pushes the `main` branch to origin.
3. Pulls the repository on the target machine.
4. Restarts the podman stack (Dankclaw only. The Dankserver media stack
runs as bare containers).
5. Relinks the LLM systemd units, reloads systemd, enables, and restarts
the active LLM service.
6. Waits for the Caddy homepage and the LLM health endpoint (over ssh).
## Manual recovery
Set `FRANK_SUDO_PW` when running headless on a host without passwordless
sudo, so that remote sudo reads the password from stdin (`sudo -S`). The
password is never stored in this repo. Both fleet hosts currently allow
passwordless sudo for the deploy user (verified 2026-08-22). The
passwordless restart helpers from the retired topology are obsolete and are
not used.
If the LLM handler fails after the target pull:
Health endpoints:
1. Inspect `journalctl -u frank-default-small -n 200` or
`journalctl -u frank-default-large -n 200`.
2. Confirm the model files, custom runtime binary, and chat template are
readable by `dank`.
3. Retry only the affected service:
- Dankserver: `http://127.0.0.1:7770/health`
- Dankclaw: `http://127.0.0.1:7770/health`
```bash
./scripts/deploy.sh dankserver --service frank-default-small
# or
./scripts/deploy.sh dankclaw --service frank-default-large
```
## Manual Update
4. Verify the matching health endpoint before declaring recovery.
### Update Dankserver (frank-default-small)
Run these commands on Dankserver:
```
cd /home/dank/serverAppConfigs
git pull origin main
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-small
curl -sf http://127.0.0.1:7770/health
```
### Update Dankclaw (frank-default-large)
Run these commands on Dankclaw:
```
cd /home/dank/serverAppConfigs
git pull origin main
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:7770/health
```
## Service Units
### Dankserver — frank-default-small
The active service on Dankserver is upstream llama.cpp (b10121) serving
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small`
on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server`
binary with expert tensors selected by an explicit `--override-tensor`
regex. Expert tensors from every full-attention block and blocks 35 through 40
stay on CUDA; other expert tensors are CPU-resident. The service uses 7 CPU
threads, ngram-mod speculative decoding only, f16 KV, context 491520 (3 slots
of 163840 each), batch
4096, ubatch 1024, and `--no-mmap`. The unit is
`scripts/frank-default-small.service`. It launches through
`scripts/serve-frank-small.sh`, which reads tunables from
`scripts/frank-default-small.env`. It uses `ubatch=1024` and the chat template
from `peculiar-ragdoll/Qwen-Sharp-Chat-Templates`. See [[dealign-ornith-crack]].
### Dankclaw — frank-default-large
The active service on Dankclaw uses the ROCmFPX llama.cpp runtime to serve
[kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF](https://huggingface.co/kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF)
under the alias `frank-default-large` on port 7770. The main model is
`Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf`; the standalone MTP draft is
`mtp-Qwen3.8-27B-Q4_0.gguf`, and the Q8_0 vision projector is loaded alongside
it. The K-side KV cache remains `f16`; the V-side cache uses `turbo4`. The
service uses the 262144 context / 1-slot profile. The unit file is
`scripts/frank-default-large.service`; it launches through
`scripts/serve-frank-large.sh`, which reads tunables from
`scripts/frank-default-large.env`, applying the Qwen-Sharp chat template.
> The Q8_0 AGENT experiment was retired on 2026-09-01 after repeated
> `set_rows` crashes and device-local allocation failures. The former caf-eemin
> ROCmI4 embedded-MTP model and the v3 PromptForge payload are historical
> deployments. The intermediate DFlash2 Vulkan deployment was retired on
> 2026-08-21. See [[kingjones-q8agent-rocmfpx]] and [[dflash2-large-vulkan]].
## Validation
The deploy script checks the health endpoint after a restart. For deeper
checks, run a completion request against the endpoint:
```
curl -s http://127.0.0.1:7770/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"frank-default-small","messages":[{"role":"user","content":"Say OK"}]}'
```
## Troubleshooting
If the health check fails after a deploy:
1. Read the service logs with `journalctl -u <service> -n 200`.
2. Confirm that the model files exist in the cache.
3. Confirm that the GPU has free memory (`nvidia-smi` on Dankserver, `rocm-smi` on Dankclaw).
4. Check the service status with `systemctl status <service>`.
## Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13)
On 2026-08-13 the old `frank-default-large` service crash-looped for hours
on Dankclaw. Root cause: the env files exported `TEMP=1.0` (a sampling
default). The ROCm HIP runtime reads `TEMP` as its JIT kernel-cache
directory, used the path `1.0`, failed, and segfaulted on a NULL object.
Fix: rename the sampling variable to `TEMPERATURE` (commit `227ce01`).
If a service segfaults at load with no obvious cause, compare the exported
env against the HIP runtime's env reads. Do not name any tunable `TEMP`,
`HOST`, or `GPU_MAX_HEAP_SIZE`.
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
Historical: on the retired topology, parallel jobs on the Dankserver 35B
service died with a cuBLAS crash under researcher load. The affected
service is retired. A client abort logs `W srv stop: cancel task,
id_task = N`. A natural stop logs only `slot release` with
`truncated = 0`. The server never preempts a running generation.
## Incident: frank-default-small Crash Loop After Distro Upgrade (2026-08-21)
On 2026-08-21 a release upgrade moved Dankserver from Ubuntu 24.04 to
Ubuntu 26.04 (Resolute Raccoon). The upgrade removed the CUDA 12 runtime
packages: `libcudart12`, `libcublas12`, `libcublaslt12`, and
`nvidia-cuda-toolkit`. The `llama-server` binary links
`libcudart.so.12` and `libcublas.so.12`. On every start the binary exited
with code 127. systemd restart-looped the service continuously. The health
endpoint on port 7770 stayed down.
Fix: reinstall the runtime packages. The same versions remain in the 26.04
repository.
sudo apt-get install -y libcudart12 libcublas12 libcublaslt12
Verify that the binary resolves all libraries:
ldd /home/dank/llama.cpp/build/bin/llama-server | grep "not found"
Start the service and check the health endpoint on port 7770.
After any release upgrade on Dankserver, run the `ldd` check above. A
missing CUDA 12 library means the packages need a reinstall.
Do not stop the other host's inference service during recovery. For benchmark
work, follow the separate stop/benchmark/restart procedure in [[Testing Policy]]
and the repository rules.
## See Also
- [AGENTS.md](../../AGENTS.md) — the git workflow rules
- [README.md](../../README.md) — repository overview
- [index.md](index.md) — LLM fleet knowledge base index
- [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) — the full redo runbook
- [docs/index.md](../index.md) — the project knowledge base (podman stacks)
- [Deployment Flow](../Architecture/Deployment%20Flow.md)
- [[index]]
- [Dankclaw deployment architecture](../Architecture/Network%20Architecture.md)
docs/LLM/index.md
+2 -2
View File
@@ -120,5 +120,5 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
- [Project knowledge base](../index.md) — the Podman app stacks on the same
two hosts. OpenWebUI consumes the `:7770` endpoint on Dankserver. The
Gitea instance on Dankclaw hosts this repository.
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the unified
git-based deploy for both workloads
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
git-based deploy with inference isolation
docs/index.md
+1 -1
View File
@@ -27,7 +27,7 @@ covers both workloads).
- [[Network Architecture]] — Caddy reverse proxy, hostname routing, Gluetun VPN path
- [[Storage Policy]] — bind mounts, data locations, ownership
- [[Deployment Flow]] — git-based deploy, target map, health checks
- [[Deployment Flow]] — service-scoped git deploy, target map, and health checks
- [[Testing Policy]] — the test seams and how to run them
## Dankserver Storage