dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Deploy services independently by changed paths

Browse Source
This commit is contained in:
Frank
2026-08-25 20:13:33 -07:00
parent 2277a2e9ac
commit 1ba47e4d22
18 changed files with 1462 additions and 894 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+26 -17
View File
@@ -46,8 +46,9 @@ Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`.
- `dankserver/` — compose.yaml + Caddyfile + .env.example for the media stack - `dankserver/` — compose.yaml + Caddyfile + .env.example for the media stack
- `dankclaw/` — compose.yaml + Caddyfile + .env.example for the AI stack - `dankclaw/` — compose.yaml + Caddyfile + .env.example for the AI stack
- `shared/` — homepage, Prometheus, Loki, Alloy, SearXNG configs - `shared/` — homepage, Prometheus, Loki, Alloy, SearXNG configs
- `scripts/` — `deploy.sh` (unified deployer), LLM serve scripts, systemd - `scripts/` — `deploy.sh`, `deploy-plan.py`, and `deploy-remote.sh`
units, env tunables, bench harnesses (service-scoped deployer), LLM serve scripts, systemd units, env tunables,
and bench harnesses
- `old/` — retired PP/NPU-era LLM scripts (reference only) - `old/` — retired PP/NPU-era LLM scripts (reference only)
- `docs/` — knowledge base (Obsidian vault). `docs/LLM/` is the LLM fleet - `docs/` — knowledge base (Obsidian vault). `docs/LLM/` is the LLM fleet
- `tests/` — bats test suite - `tests/` — bats test suite
@@ -92,20 +93,28 @@ All edits flow through git. Do not edit files on the remote machine directly.
4. **Run the tests** — `./tests/run.sh` must pass 4. **Run the tests** — `./tests/run.sh` must pass
5. **Commit and push** — push to origin (Gitea on Dankclaw) 5. **Commit and push** — push to origin (Gitea on Dankclaw)
6. **Pull on the target** — `git pull` on the remote machine 6. **Pull on the target** — `git pull` on the remote machine
7. **Restart** — `scripts/deploy.sh` pulls, restarts the stack and the LLM 7. **Deploy** — `scripts/deploy.sh` pulls, plans, and dispatches only the
service, and checks health changed service(s), then checks the selected service health
## Deployment ## Deployment
Use `scripts/deploy.sh` to deploy committed changes to a target. One script Use `scripts/deploy.sh` to deploy committed changes to a target. The planner
covers both workloads per host: the podman stack and the LLM service. maps changed paths to individual services; the remote runner restarts only the
selected service and records its successful revision outside the checkout.
``` ```
./scripts/deploy.sh dankclaw # AI stack + frank-default-large (:7770) ./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankserver # media stack + frank-default-small (:7770) ./scripts/deploy.sh dankserver
./scripts/deploy.sh dankclaw --dry-run # preview without changes ./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-default-small
./scripts/deploy.sh dankclaw --dry-run
``` ```
A documentation-only change is a no-op. The LLM services are selected only by
their own unit/env/serve files or the shared chat template, so unrelated
container changes cannot interrupt inference. `frank-default-npu` is relink-only
when explicitly selected.
Set `FRANK_SUDO_PW` for headless runs on a host without passwordless sudo Set `FRANK_SUDO_PW` for headless runs on a host without passwordless sudo
(remote sudo reads it from stdin via `sudo -S`). The value is never stored (remote sudo reads it from stdin via `sudo -S`). The value is never stored
in this repo. in this repo.
@@ -115,10 +124,10 @@ graph LR
A[Edit locally] --> B[Tests pass] A[Edit locally] --> B[Tests pass]
B --> C[Commit] B --> C[Commit]
C --> D[Push to origin] C --> D[Push to origin]
D --> E[deploy.sh: pull on target] D --> E[deploy.sh computes service plan]
E --> F[restart podman stack] E --> F[Target pulls with ff-only]
F --> G[relink + restart LLM unit] F --> G[deploy-remote.sh dispatches selected service]
G --> H[health: homepage + LLM] G --> H[Per-service health check]
``` ```
## Testing ## Testing
@@ -133,8 +142,8 @@ development machine.
Test seams: compose validity, env consistency (no leaked secrets), Caddyfile Test seams: compose validity, env consistency (no leaked secrets), Caddyfile
validity, quadlet units, docs completeness, script syntax (all of `scripts/`, validity, quadlet units, docs completeness, script syntax (all of `scripts/`,
including the LLM serve scripts), LLM unit integrity (units point at files in including the LLM serve scripts), LLM unit integrity (units point at files in
this repo), LLM secret hygiene, deployment flow. See this repo), LLM secret hygiene, deployment flow, and service-scoped deployment
[docs/Architecture/Testing Policy.md](docs/Architecture/Testing%20Policy.md). planning/dispatch. See [docs/Architecture/Testing Policy.md](docs/Architecture/Testing%20Policy.md).
## Secret Rules ## Secret Rules
@@ -387,7 +396,7 @@ Direct edits on the remote drift from version control. Future agents will not
know what changed. Git is the single source of truth. The two workloads share know what changed. Git is the single source of truth. The two workloads share
two hosts, one Caddy, and one Gitea. One repo keeps them in one history. two hosts, one Caddy, and one Gitea. One repo keeps them in one history.
> User request → Fetch from remote → Commit here → Edit → Test → Commit → Push → Pull on remote → Deploy + restart > User request → Fetch from remote → Commit here → Edit → Test → Commit → Push → Pull on remote → Dispatch changed service(s) → Health
## Summary ## Summary
@@ -396,7 +405,7 @@ two hosts, one Caddy, and one Gitea. One repo keeps them in one history.
| Editing a compose/Caddyfile/serve script | **Work through git** — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote directly. | | Editing a compose/Caddyfile/serve script | **Work through git** — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote directly. |
| Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before making changes. | | Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before making changes. |
| Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. | | Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. |
| Deploying a committed change | **Run `scripts/deploy.sh <target>`** — it pushes, pulls on the target, restarts the stack and the LLM service, and checks both health endpoints. See [docs/LLM/deployment.md](docs/LLM/deployment.md). | | Deploying a committed change | **Run `scripts/deploy.sh <target>`** — it pushes, pulls with fast-forward-only, plans changed paths, dispatches selected services, and checks their health. See [docs/LLM/deployment.md](docs/LLM/deployment.md). |
| Benchmarking | **Stop the service first** — scratch port, restart, verify health. See the Benchmarking Policy above. | | Benchmarking | **Stop the service first** — scratch port, restart, verify health. See the Benchmarking Policy above. |
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. | | Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. | | Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
README.md
+5 -4
View File
@@ -37,7 +37,7 @@ one deploy script.
- `dankserver/` — compose.yaml + Caddyfile + .env.example for the Dankserver stack - `dankserver/` — compose.yaml + Caddyfile + .env.example for the Dankserver stack
- `dankclaw/` — compose.yaml + Caddyfile + .env.example for the Dankclaw stack - `dankclaw/` — compose.yaml + Caddyfile + .env.example for the Dankclaw stack
- `shared/` — homepage, prometheus config, deploy helpers - `shared/` — homepage, prometheus config, deploy helpers
- `scripts/` — `deploy.sh` (unified: stack + LLM), LLM serve scripts, systemd units, env tunables, bench harnesses - `scripts/` — `deploy.sh` and `deploy-remote.sh` (service-scoped deployment), the planner, LLM serve scripts, systemd units, env tunables, and bench harnesses
- `old/` — retired PP/NPU-era LLM scripts (reference only) - `old/` — retired PP/NPU-era LLM scripts (reference only)
- `docs/` — project knowledge base (Obsidian vault, wikilinks). `docs/LLM/` is the LLM fleet - `docs/` — project knowledge base (Obsidian vault, wikilinks). `docs/LLM/` is the LLM fleet
- `tests/` — bats test suite (see [[Testing Policy]]) - `tests/` — bats test suite (see [[Testing Policy]])
@@ -46,9 +46,10 @@ one deploy script.
See [docs/Architecture/Deployment Flow.md](docs/Architecture/Deployment%20Flow.md) See [docs/Architecture/Deployment Flow.md](docs/Architecture/Deployment%20Flow.md)
and [docs/LLM/deployment.md](docs/LLM/deployment.md). Run and [docs/LLM/deployment.md](docs/LLM/deployment.md). Run
`./scripts/deploy.sh <dankclaw|dankserver>` after pushing committed changes. `./scripts/deploy.sh <dankclaw|dankserver>` after committing changes. It
It pushes, pulls on the target, restarts the podman stack and the LLM plans and deploys only the changed services. Use `--service NAME` for an
service, and checks both health endpoints. explicit single-service retry; inference is restarted only when its own files
(or the shared chat template) changed.
## Knowledge Base ## Knowledge Base
docs/Architecture/Deployment Flow.md
+99 -30
View File
@@ -1,60 +1,129 @@
---
type: guide
title: Deployment Flow
description: Service-scoped Git deployment for the two homelab hosts.
tags:
- deployment
- operations
- git
timestamp: 2026-09-03
---
# Deployment Flow # Deployment Flow
Git is the single source of truth. A `deploy.sh` script moves committed Git is the single source of truth. `scripts/deploy.sh` pushes the committed
changes to the target hosts over SSH. One run per target covers both `main` branch, pulls it on one target, and applies only the runtime services
workloads on that host: the podman stack and the LLM service. whose tracked inputs changed. A SearXNG change no longer recreates the
inference service; an LLM change no longer restarts unrelated containers.
## Workflow ## Workflow
```mermaid ```mermaid
graph LR graph LR
A[Edit in this repo] --> B[Commit] A[Edit in this repo] --> B[Run tests]
B --> C[Push to origin] B --> C[Commit main]
C --> D[deploy.sh: pull on target] C --> D[deploy.sh computes plan]
D --> E[restart stack] D --> E[Push main]
E --> F[health check] E --> F[Target pulls with ff-only]
F --> G[deploy-remote.sh dispatches selected services]
G --> H[Per-service health check]
``` ```
1. Edit files locally in this repository. 1. Edit files locally in this repository.
2. Commit and push. 2. Run `./tests/run.sh`, then commit the complete change on `main`.
3. Run `./scripts/deploy.sh <dankclaw|dankserver>`. 3. Run `./scripts/deploy.sh <target>`.
4. The script pulls the repo on the target and restarts the podman stack 4. The planner maps changed paths to services. Compose YAML is compared by
and the LLM service. service definition so an edit to one service does not restart its peers.
5. It verifies service health after restart (homepage and LLM endpoint). 5. The target pulls with `git pull --ff-only origin main` and runs the remote
dispatcher with the explicit plan.
6. The dispatcher updates only selected Quadlet, compose, homepage, Caddy, or
systemd services and records each successful service revision outside Git.
## Deploy Script ## Commands
``` ```bash
./scripts/deploy.sh dankclaw # pull + restart on Dankclaw ./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankserver # pull + restart on Dankserver ./scripts/deploy.sh dankserver
./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-default-small
./scripts/deploy.sh dankclaw --dry-run ./scripts/deploy.sh dankclaw --dry-run
``` ```
SSH uses the existing key pair (`dank@dankclaw`, `dank@dankserver`). No passwords are stored or documented in this repo. If sudo is needed, use the interactive sudo on the target — never commit a password. `--service` is repeatable and is an explicit retry/override. Without it, the
planner compares the local `HEAD` to `origin/main`. A documentation-only
commit produces an empty plan. The remote state directory can also replay a
service whose earlier deployment failed, but it never expands a plan to a
service on the other host.
## Service managers
| Service family | Runtime manager | Scope behavior |
|---|---|---|
| Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
| SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. |
| `frank-default-large` | systemd + `systemctl` | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
| `frank-default-small` | systemd + `systemctl` | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
| Dankserver media inventory | host-managed bare containers | Compose is inventory only; the scoped deploy does not create duplicate containers. |
The shared chat template maps to the inference service on the selected target.
The inactive `frank-default-npu` unit is relink-only when explicitly selected;
it is not restarted by an unrelated large-model deployment.
## Caddy safety
Caddy runs on Dankclaw and fronts both hosts. The dispatcher never stops
`gitea-caddy` implicitly. If the legacy `gitea-caddy` container owns ports
80/443 while the compose `caddy` container is absent or stopped, a Caddy
selection fails closed rather than replacing the live proxy. Resolve port
ownership explicitly before retrying. A Caddy reload failure falls back to a
restart of **only** the selected `caddy` container.
Homepage files, generated `deployed.js`, the status checker, and the rendered
Caddyfile are updated only when their service is selected. A media-only
Dankserver deployment does not touch Dankclaw's proxy or homepage.
## Per-service state
The remote runner writes one revision file per successfully completed service
to:
```text
${XDG_STATE_HOME:-~/.local/state}/serverAppConfigs/deploy-state/
```
The directory is outside the checkout and is never committed. A failed handler
is not marked successful. On the next ordinary deployment, the runner compares
the recorded revision with the new checkout and retries only a still-changed
service. This prevents a failed Caddy or LLM deployment from being silently
forgotten while avoiding a host-wide restart.
## Targets ## Targets
| Target | Host | Stack | | Target | Host | Repository checkout |
|--------|------|-------| |---|---|---|
| dankclaw | dank@dankclaw (192.168.0.143) | `dankclaw/` | | dankclaw | `dank@dankclaw` (`192.168.0.143`) | `/home/dank/serverAppConfigs` |
| dankserver | dank@dankserver (192.168.0.235) | `dankserver/` | | dankserver | `dank@dankserver` (`192.168.0.235`) | `/home/dank/serverAppConfigs` |
## Health Verification SSH uses the existing key pair. No password is stored or documented in this
repository. If sudo is needed on a headless target, `FRANK_SUDO_PW` is read
The deploy script polls the health endpoint of each service after restart. Service healthchecks are defined in each compose file. A service that fails its healthcheck is reported, not hidden. from the environment and is never written to the repository.
## Tests ## Tests
Run the test suite before pushing: Run the full suite before pushing:
``` ```bash
./tests/run.sh ./tests/run.sh
``` ```
The suite validates compose YAML, env consistency, Caddyfiles, docs completeness, and script syntax. See [[Testing Policy]]. The deployment seams are in `tests/deployment_scoping.bats` and
`tests/deployment_flow.bats`. They cover path isolation, compose narrowing,
Caddy safety, inference isolation, per-service state, and legacy wrapper
routing. See [[Testing Policy]].
## See Also ## See Also
- [[Storage Policy]]
- [[Testing Policy]] - [[Testing Policy]]
- [LLM deployment docs](../LLM/deployment.md) — the LLM half of the same flow - [LLM deployment](../LLM/deployment.md)
- [[Caddy]]
- [[SearXNG]]
docs/Architecture/Testing Policy.md
+1 -1
View File
@@ -11,7 +11,7 @@ This repository tests its configuration at defined seams. The tests run with `ba
| Caddyfile validity | `tests/caddyfile.bats` | per-host Caddyfile, `.local` convention, balanced braces, basic auth present | | Caddyfile validity | `tests/caddyfile.bats` | per-host Caddyfile, `.local` convention, balanced braces, basic auth present |
| Docs completeness | `tests/docs_completeness.bats` | every compose service has a docs page; index links every page | | Docs completeness | `tests/docs_completeness.bats` | every compose service has a docs page; index links every page |
| Script syntax | `tests/scripts_syntax.bats` | `bash -n` clean, executable bits, shellcheck when available | | Script syntax | `tests/scripts_syntax.bats` | `bash -n` clean, executable bits, shellcheck when available |
| Deployment flow | `tests/deployment_flow.bats` | deploy.sh exists, targets known, SSH reachable (skips offline) | | Deployment flow | `tests/deployment_flow.bats`, `tests/deployment_scoping.bats` | scoped dispatcher exists, targets are known, service plans isolate inference, Caddy ownership is guarded, and SSH reachability is checked (skips offline) |
## Focused pre-commit testing ## Focused pre-commit testing
docs/LLM/deployment.md
+96 -199
View File
@@ -1,227 +1,124 @@
---
type: guide
title: LLM Deployment
description: Scoped deployment and update process for the two inference services.
tags:
- llm
- deployment
- systemd
timestamp: 2026-09-03
---
# Deployment and Update Process # Deployment and Update Process
This page describes how to deploy and update the LLM servers on Dankserver This page describes how to deploy and update the LLM servers on Dankserver and
and Dankclaw. The process uses git as the single source of truth. All script Dankclaw. The process uses Git as the single source of truth. A change to
edits flow through this repository. SearXNG, Caddy, or another container service does **not** restart inference.
> 2026-08-22: the former `vllmScripts` repository is merged into this repo. ## Fleet
> Both LLM services now run from the `/home/dank/serverAppConfigs` checkout
> on each host. The retired checkouts (`/home/dank/vLLM` on Dankserver,
> `/home/dank/vllmScripts` on Dankclaw) are archived on the hosts.
## Overview | Target | Host | Service | Port |
|---|---|---|---:|
| Dankserver | `dank@dankserver` | `frank-default-small` | 7770 |
| Dankclaw | `dank@dankclaw` | `frank-default-large` | 7770 |
Two servers run LLM inference services. Each server pulls its scripts from ## Scoped workflow
this repository. The deployment process moves a committed change from the
repository to a running service.
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankserver | dank@dankserver | upstream llama.cpp (Ornith-1.5-35B-A3B-CRACK Q6_K, native MTP draft, selected MoE experts offloaded via --override-tensor) | frank-default-small | 7770 |
| Dankclaw | dank@dankclaw | charlie12345/ROCmFPX llama.cpp (kingjones Qwen3.8 Uncensored Q6_0 ROCmFPX AGENT, standalone MTP, 262K ctx / 1 slot) | frank-default-large | 7770 |
## Workflow
The deployment workflow has seven steps.
```mermaid ```mermaid
graph LR graph LR
A[Edit scripts locally] --> B[Commit] A[Edit LLM files] --> B[Run tests]
B --> C[Push to origin<br>Gitea on Dankclaw] B --> C[Commit main]
C --> D[Pull on target<br>git pull] C --> D[deploy.sh selects matching service]
D --> E[Restart podman stack] D --> E[Target pulls ff-only]
E --> F[Relink + restart LLM unit] E --> F[Relink one unit]
F --> G[Verify health endpoints] F --> G[Restart one inference service]
G --> H[Wait for that service health]
``` ```
1. Fetch the current script from the target machine. Run the unified deployer from the repository root:
2. Commit the fetched version in this repository.
3. Edit the script locally in the working tree.
4. Run the tests (`./tests/run.sh` must pass).
5. Commit and push the change to origin.
6. Pull the change on the target machine.
7. Restart the services and verify that they are healthy.
## Deploy Script ```bash
./scripts/deploy.sh dankserver --service frank-default-small
The `scripts/deploy.sh` script is the unified deployer. One run per target ./scripts/deploy.sh dankclaw --service frank-default-large
covers both workloads on that host: the podman stack and the LLM service. ./scripts/deploy.sh dankclaw --dry-run
Run it from the repository root.
Run this command to deploy to Dankserver:
```
./scripts/deploy.sh dankserver
``` ```
Run this command to deploy to Dankclaw: Without `--service`, `scripts/deploy-plan.py` maps changed files to the service
on the selected host. The following files select the corresponding service:
``` - `scripts/frank-default-small.env`, `frank-default-small.service`, and
./scripts/deploy.sh dankclaw `serve-frank-small.sh` select `frank-default-small` on Dankserver.
- `scripts/frank-default-large.env`, `frank-default-large.service`, and
`serve-frank-large.sh` select `frank-default-large` on Dankclaw.
- `scripts/chat-template.jinja` selects the LLM on the selected host only.
- Documentation, tests, deployment tooling, and unrelated container changes
do not select either inference service.
The target-side `scripts/deploy-remote.sh` relinks the selected unit into
`/etc/systemd/system/`, reloads systemd, enables it, restarts it, and waits for
`http://127.0.0.1:7770/health`. The wait budget is 20 minutes because cold model
loads can be slow. A failed handler does not advance its per-service state.
## Unit and artifact paths
Both units point into the stable target checkout:
```text
/home/dank/serverAppConfigs/scripts/frank-default-small.service
/home/dank/serverAppConfigs/scripts/frank-default-large.service
``` ```
Add the `--dry-run` flag to preview without making changes: The unit files and serve scripts validate their model/runtime artifacts before
starting. Keep `REMOTE_DIR` stable; moving the checkout requires updating both
the unit absolute paths and the deployment tests in the same change.
``` ## NPU unit
./scripts/deploy.sh dankserver --dry-run
`frank-default-npu` is an inactive, historical Dankclaw endpoint. It is
relinked only when explicitly selected with `--service frank-default-npu`; the
large-model handler never restarts it. This is intentional: the NPU is not part
of the active inference path and should not be disturbed by a large-model
change.
## Isolation guarantee
The remote runner has one explicit service dispatch. It does not run a
host-wide LLM loop, `podman-compose up` for unrelated services, or a blanket
systemd restart. Each successful service is recorded under the target user's
state directory outside Git:
```text
~/.local/state/serverAppConfigs/deploy-state/
``` ```
The script does these actions: On a later ordinary deployment, a service remains pending only while its
recorded revision differs from the checkout and the planner still finds a
change in that service's inputs.
1. Verifies that the local working tree has no uncommitted changes. ## Manual recovery
2. Pushes the `main` branch to origin.
3. Pulls the repository on the target machine.
4. Restarts the podman stack (Dankclaw only. The Dankserver media stack
runs as bare containers).
5. Relinks the LLM systemd units, reloads systemd, enables, and restarts
the active LLM service.
6. Waits for the Caddy homepage and the LLM health endpoint (over ssh).
Set `FRANK_SUDO_PW` when running headless on a host without passwordless If the LLM handler fails after the target pull:
sudo, so that remote sudo reads the password from stdin (`sudo -S`). The
password is never stored in this repo. Both fleet hosts currently allow
passwordless sudo for the deploy user (verified 2026-08-22). The
passwordless restart helpers from the retired topology are obsolete and are
not used.
Health endpoints: 1. Inspect `journalctl -u frank-default-small -n 200` or
`journalctl -u frank-default-large -n 200`.
2. Confirm the model files, custom runtime binary, and chat template are
readable by `dank`.
3. Retry only the affected service:
- Dankserver: `http://127.0.0.1:7770/health` ```bash
- Dankclaw: `http://127.0.0.1:7770/health` ./scripts/deploy.sh dankserver --service frank-default-small
# or
./scripts/deploy.sh dankclaw --service frank-default-large
```
## Manual Update 4. Verify the matching health endpoint before declaring recovery.
### Update Dankserver (frank-default-small) Do not stop the other host's inference service during recovery. For benchmark
work, follow the separate stop/benchmark/restart procedure in [[Testing Policy]]
Run these commands on Dankserver: and the repository rules.
```
cd /home/dank/serverAppConfigs
git pull origin main
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-small
curl -sf http://127.0.0.1:7770/health
```
### Update Dankclaw (frank-default-large)
Run these commands on Dankclaw:
```
cd /home/dank/serverAppConfigs
git pull origin main
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:7770/health
```
## Service Units
### Dankserver — frank-default-small
The active service on Dankserver is upstream llama.cpp (b10121) serving
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small`
on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server`
binary with expert tensors selected by an explicit `--override-tensor`
regex. Expert tensors from every full-attention block and blocks 35 through 40
stay on CUDA; other expert tensors are CPU-resident. The service uses 7 CPU
threads, ngram-mod speculative decoding only, f16 KV, context 491520 (3 slots
of 163840 each), batch
4096, ubatch 1024, and `--no-mmap`. The unit is
`scripts/frank-default-small.service`. It launches through
`scripts/serve-frank-small.sh`, which reads tunables from
`scripts/frank-default-small.env`. It uses `ubatch=1024` and the chat template
from `peculiar-ragdoll/Qwen-Sharp-Chat-Templates`. See [[dealign-ornith-crack]].
### Dankclaw — frank-default-large
The active service on Dankclaw uses the ROCmFPX llama.cpp runtime to serve
[kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF](https://huggingface.co/kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF)
under the alias `frank-default-large` on port 7770. The main model is
`Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf`; the standalone MTP draft is
`mtp-Qwen3.8-27B-Q4_0.gguf`, and the Q8_0 vision projector is loaded alongside
it. The K-side KV cache remains `f16`; the V-side cache uses `turbo4`. The
service uses the 262144 context / 1-slot profile. The unit file is
`scripts/frank-default-large.service`; it launches through
`scripts/serve-frank-large.sh`, which reads tunables from
`scripts/frank-default-large.env`, applying the Qwen-Sharp chat template.
> The Q8_0 AGENT experiment was retired on 2026-09-01 after repeated
> `set_rows` crashes and device-local allocation failures. The former caf-eemin
> ROCmI4 embedded-MTP model and the v3 PromptForge payload are historical
> deployments. The intermediate DFlash2 Vulkan deployment was retired on
> 2026-08-21. See [[kingjones-q8agent-rocmfpx]] and [[dflash2-large-vulkan]].
## Validation
The deploy script checks the health endpoint after a restart. For deeper
checks, run a completion request against the endpoint:
```
curl -s http://127.0.0.1:7770/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"frank-default-small","messages":[{"role":"user","content":"Say OK"}]}'
```
## Troubleshooting
If the health check fails after a deploy:
1. Read the service logs with `journalctl -u <service> -n 200`.
2. Confirm that the model files exist in the cache.
3. Confirm that the GPU has free memory (`nvidia-smi` on Dankserver, `rocm-smi` on Dankclaw).
4. Check the service status with `systemctl status <service>`.
## Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13)
On 2026-08-13 the old `frank-default-large` service crash-looped for hours
on Dankclaw. Root cause: the env files exported `TEMP=1.0` (a sampling
default). The ROCm HIP runtime reads `TEMP` as its JIT kernel-cache
directory, used the path `1.0`, failed, and segfaulted on a NULL object.
Fix: rename the sampling variable to `TEMPERATURE` (commit `227ce01`).
If a service segfaults at load with no obvious cause, compare the exported
env against the HIP runtime's env reads. Do not name any tunable `TEMP`,
`HOST`, or `GPU_MAX_HEAP_SIZE`.
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
Historical: on the retired topology, parallel jobs on the Dankserver 35B
service died with a cuBLAS crash under researcher load. The affected
service is retired. A client abort logs `W srv stop: cancel task,
id_task = N`. A natural stop logs only `slot release` with
`truncated = 0`. The server never preempts a running generation.
## Incident: frank-default-small Crash Loop After Distro Upgrade (2026-08-21)
On 2026-08-21 a release upgrade moved Dankserver from Ubuntu 24.04 to
Ubuntu 26.04 (Resolute Raccoon). The upgrade removed the CUDA 12 runtime
packages: `libcudart12`, `libcublas12`, `libcublaslt12`, and
`nvidia-cuda-toolkit`. The `llama-server` binary links
`libcudart.so.12` and `libcublas.so.12`. On every start the binary exited
with code 127. systemd restart-looped the service continuously. The health
endpoint on port 7770 stayed down.
Fix: reinstall the runtime packages. The same versions remain in the 26.04
repository.
sudo apt-get install -y libcudart12 libcublas12 libcublaslt12
Verify that the binary resolves all libraries:
ldd /home/dank/llama.cpp/build/bin/llama-server | grep "not found"
Start the service and check the health endpoint on port 7770.
After any release upgrade on Dankserver, run the `ldd` check above. A
missing CUDA 12 library means the packages need a reinstall.
## See Also ## See Also
- [AGENTS.md](../../AGENTS.md) — the git workflow rules - [Deployment Flow](../Architecture/Deployment%20Flow.md)
- [README.md](../../README.md) — repository overview - [[index]]
- [index.md](index.md) — LLM fleet knowledge base index - [Dankclaw deployment architecture](../Architecture/Network%20Architecture.md)
- [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) — the full redo runbook
- [docs/index.md](../index.md) — the project knowledge base (podman stacks)
docs/LLM/index.md
+2 -2
View File
@@ -120,5 +120,5 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
- [Project knowledge base](../index.md) — the Podman app stacks on the same - [Project knowledge base](../index.md) — the Podman app stacks on the same
two hosts. OpenWebUI consumes the `:7770` endpoint on Dankserver. The two hosts. OpenWebUI consumes the `:7770` endpoint on Dankserver. The
Gitea instance on Dankclaw hosts this repository. Gitea instance on Dankclaw hosts this repository.
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the unified - [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
git-based deploy for both workloads git-based deploy with inference isolation
docs/index.md
+1 -1
View File
@@ -27,7 +27,7 @@ covers both workloads).
- [[Network Architecture]] — Caddy reverse proxy, hostname routing, Gluetun VPN path - [[Network Architecture]] — Caddy reverse proxy, hostname routing, Gluetun VPN path
- [[Storage Policy]] — bind mounts, data locations, ownership - [[Storage Policy]] — bind mounts, data locations, ownership
- [[Deployment Flow]] — git-based deploy, target map, health checks - [[Deployment Flow]] — service-scoped git deploy, target map, and health checks
- [[Testing Policy]] — the test seams and how to run them - [[Testing Policy]] — the test seams and how to run them
## Dankserver Storage ## Dankserver Storage
scripts/deploy-open-webui-quadlet.sh
+10 -100
View File
@@ -1,104 +1,14 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# deploy-open-webui-quadlet.sh — Install and (re)start open-webui on dankclaw. #!/usr/bin/env bash
# # Compatibility wrapper for the service-scoped deployment dispatcher.
# open-webui is boot-persistent via Quadlet (see AGENTS.md). This script:
# 1. Pushes main to origin
# 2. Pulls the repo on the target
# 3. Creates the data dir and copies the unit + network into ~/.config/containers/systemd/
# 4. daemon-reloads and starts open-webui (first run pulls the image)
#
# It is idempotent: it stops any running unit and removes any container with the
# name (a prior run or a bare-container leftover) before reinstalling and
# starting. The data dir is preserved across reruns, so existing chats survive.
#
# Usage:
# ./scripts/deploy-open-webui-quadlet.sh <target> [--dry-run]
# ./scripts/deploy-open-webui-quadlet.sh dankclaw
set -euo pipefail set -euo pipefail
REMOTE_USER="dank"
REMOTE_HOST="dankclaw"
REMOTE_DIR="/home/dank/serverAppConfigs"
QDIR="dankclaw/open-webui/quadlet"
UNIT="open-webui.service"
DATA_DIR="/home/dank/Containers/open-webui/data"
TARGET="${1:-}" TARGET="${1:-}"
DRY_RUN=0 shift || true
if [ "${2:-}" = "--dry-run" ]; then case "$TARGET" in
DRY_RUN=1 dankclaw) exec "$(dirname "$0")/deploy.sh" dankclaw --service open-webui "$@" ;;
fi *)
echo "This stack runs only on dankclaw (got: ${TARGET:-<missing>})" >&2
if [ -z "${TARGET:-}" ]; then exit 2
echo "Usage: $0 <target> [--dry-run]" ;;
exit 1 esac
fi
if [ "${TARGET}" != "dankclaw" ]; then
echo "This stack runs only on dankclaw (got: ${TARGET})"
exit 1
fi
# ---------------------------------------------------------------------------
# Local pre-flight
# ---------------------------------------------------------------------------
cd "$(dirname "$0")/.."
if ! git diff --quiet HEAD; then
echo "FAIL: Local working tree has uncommitted changes. Commit first."
exit 1
fi
if [ "$DRY_RUN" = "1" ]; then
echo "[dry-run] Would push main, pull on ${REMOTE_HOST}, install ${UNIT}, and start it (data at ${DATA_DIR})."
exit 0
fi
echo "=== Deploying open-webui quadlet to ${REMOTE_HOST} ==="
echo "[1/3] Pushing local main to origin..."
git push origin main
REMOTE="${REMOTE_USER}@${REMOTE_HOST}"
echo "[2/3] Installing and starting ${UNIT} on ${REMOTE_HOST}..."
ssh "$REMOTE" bash -s <<REMOTE_EOF
set -euo pipefail
cd '${REMOTE_DIR}'
echo ' Pulling repo...'
git pull origin main
echo ' Stopping any running unit (idempotent)...'
systemctl --user stop '${UNIT}' 2>/dev/null || true
echo ' Removing any container with the name (prior run or bare-container leftover)...'
podman rm -f open-webui 2>/dev/null || true
echo ' Creating the data dir (preserved across reruns)...'
mkdir -p '${DATA_DIR}'
echo ' Copying the quadlet unit + network into ~/.config/containers/systemd/ ...'
mkdir -p ~/.config/containers/systemd
cp ${QDIR}/open-webui.container ~/.config/containers/systemd/
cp ${QDIR}/open-webui.network ~/.config/containers/systemd/
echo ' daemon-reload (generates the unit)...'
# Quadlet units are generator-managed: systemctl enable fails on them. The
# [Install] WantedBy=default.target in the .container file is itself the
# enablement (is-enabled reports "generated"); with user linger on it starts at
# boot. So: daemon-reload, then start.
systemctl --user daemon-reload
echo ' Starting (first run pulls the image; this can take a while)...'
systemctl --user start '${UNIT}'
echo ' Unit state:'
systemctl --user --no-pager status '${UNIT}' 2>&1 | grep -E "●|Active:" || true
REMOTE_EOF
echo "[3/3] Verifying open-webui is up..."
ssh "$REMOTE" '
echo " is-enabled: $(systemctl --user is-enabled open-webui.service 2>&1) (boot-persistent)"
echo " port 3005:"; (ss -tlnp 2>/dev/null || netstat -tlnp 2>/dev/null) | grep -E ":3005\b" || echo " (not listening yet)"
echo " HTTP probe:"; curl -s -o /dev/null -w " http://127.0.0.1:3005 -> %{http_code}\n" -m 10 http://127.0.0.1:3005/ || echo " (no HTTP yet)"
' || true
echo "=== open-webui quadlet deployed to ${REMOTE_HOST} ==="
scripts/deploy-pihole-quadlet.sh
+9 -114
View File
@@ -1,118 +1,13 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# deploy-pihole-quadlet.sh — Install and (re)start the Pi-hole LAN DNS on # Compatibility wrapper for the service-scoped deployment dispatcher.
# dankclaw. Pi-hole retired the dnsmasq quadlet (2026-08-24).
#
# Pi-hole is boot-persistent via Quadlet (see AGENTS.md). This script:
# 1. Pushes main to origin
# 2. Pulls the repo on the target
# 3. Creates the data dirs and copies the quadlet unit + env + addn-hosts
# 4. daemon-reloads and starts pihole (first run pulls the image)
#
# It is idempotent: it stops any running unit and removes any container with
# the name (a prior run or a bare-container leftover) before reinstalling and
# starting. The data dirs (etc-pihole, etc-dockers) are preserved across
# reruns, so the blocklist and web settings survive. The env file is created
# from env.example only if absent; fill the real WEBPASSWORD before the web
# UI is usable.
#
# Usage:
# ./scripts/deploy-pihole-quadlet.sh <target> [--dry-run]
# ./scripts/deploy-pihole-quadlet.sh dankclaw
set -euo pipefail set -euo pipefail
REMOTE_USER="dank"
REMOTE_HOST="dankclaw"
REMOTE_DIR="/home/dank/serverAppConfigs"
QDIR="dankclaw/pihole"
UNIT="pihole.service"
DATA_DIR="/home/dank/Containers/pihole"
TARGET="${1:-}" TARGET="${1:-}"
DRY_RUN=0 shift || true
if [ "${2:-}" = "--dry-run" ]; then case "$TARGET" in
DRY_RUN=1 dankclaw) exec "$(dirname "$0")/deploy.sh" dankclaw --service pihole "$@" ;;
fi *)
echo "This stack runs only on dankclaw (got: ${TARGET:-<missing>})" >&2
if [ -z "${TARGET:-}" ]; then exit 2
echo "Usage: $0 <target> [--dry-run]" ;;
exit 1 esac
fi
if [ "${TARGET}" != "dankclaw" ]; then
echo "This stack runs only on dankclaw (got: ${TARGET})"
exit 1
fi
# ---------------------------------------------------------------------------
# Local pre-flight
# ---------------------------------------------------------------------------
cd "$(dirname "$0")/.."
if ! git diff --quiet HEAD; then
echo "FAIL: Local working tree has uncommitted changes. Commit first."
exit 1
fi
if [ "$DRY_RUN" = "1" ]; then
echo "[dry-run] Would push main, pull on ${REMOTE_HOST}, install ${UNIT}, and start it (data at ${DATA_DIR})."
exit 0
fi
echo "=== Deploying pihole quadlet to ${REMOTE_HOST} ==="
echo "[1/3] Pushing local main to origin..."
git push origin main
REMOTE="${REMOTE_USER}@${REMOTE_HOST}"
echo "[2/3] Installing and starting ${UNIT} on ${REMOTE_HOST}..."
ssh "$REMOTE" bash -s <<REMOTE_EOF
set -euo pipefail
cd '${REMOTE_DIR}'
echo ' Pulling repo...'
git pull origin main
echo ' Stopping any running unit (idempotent)...'
systemctl --user stop '${UNIT}' 2>/dev/null || true
echo ' Removing any container with the name (prior run or bare-container leftover)...'
podman rm -f pihole 2>/dev/null || true
echo ' Creating the data dirs (preserved across reruns)...'
mkdir -p '${DATA_DIR}/etc-pihole' '${DATA_DIR}/etc-dockers'
echo ' Creating the env file from env.example if absent...'
if [ ! -f '${DATA_DIR}/env' ]; then
cp ${QDIR}/env.example '${DATA_DIR}/env'
echo ' env created: set the real WEBPASSWORD before the web UI is usable'
else
echo ' env present, keeping it'
fi
echo ' Copying the record file (addn-hosts)...'
cp ${QDIR}/addn-hosts '${DATA_DIR}/addn-hosts'
echo ' Copying the quadlet unit into ~/.config/containers/systemd/ ...'
mkdir -p ~/.config/containers/systemd
cp ${QDIR}/quadlet/pihole.container ~/.config/containers/systemd/
echo ' daemon-reload (generates the unit)...'
# Quadlet units are generator-managed: systemctl enable fails on them. The
# [Install] WantedBy=default.target in the .container file is itself the
# enablement (is-enabled reports "generated"); with user linger on it starts
# at boot. So: daemon-reload, then start.
systemctl --user daemon-reload
echo ' Starting (first run pulls the image; this can take a while)...'
systemctl --user start '${UNIT}'
echo ' Unit state:'
systemctl --user --no-pager status '${UNIT}' 2>&1 | grep -E "●|Active:" || true
REMOTE_EOF
echo "[3/3] Verifying pihole is up..."
ssh "$REMOTE" '
echo " is-enabled: $(systemctl --user is-enabled pihole.service 2>&1) (boot-persistent)"
echo " port 53:"; (ss -ulpn 2>/dev/null) | grep -E ":53\b" || echo " (not listening yet)"
echo " resolution:"; dig @127.0.0.1 romm.franksdank.cloud +short 2>/dev/null || echo " (no answer yet)"
' || true
echo "=== pihole quadlet deployed to ${REMOTE_HOST} ==="
scripts/deploy-plan.py
Executable
+353
View File
@@ -0,0 +1,353 @@
#!/usr/bin/env python3
"""Plan runtime services affected by repository changes.
The planner is intentionally side-effect free. It accepts either an explicit
path list or two Git revisions. The latter lets compose-file changes resolve
to only the service definitions that actually changed instead of restarting an
entire compose project.
"""
from __future__ import annotations
import argparse
import subprocess
import sys
from pathlib import PurePosixPath
from typing import Any
import yaml
# Dependency-sensitive services come before their consumers. Independent
# compose services follow. LLM services are deliberately last and are never
# implied by a generic host, stack, documentation, or config change.
SERVICE_ORDER = (
"gluetun",
"searxng-valkey",
"searxng",
"pihole",
"open-webui",
"gitea-postgres",
"gitea-server",
"gitea-caddy",
"comfyui",
"hermes",
"hermes-dashboard",
"prometheus",
"grafana",
"loki",
"alloy",
"node-exporter",
"authentik-postgres",
"authentik-server",
"authentik-worker",
"caddy",
"homepage",
"mariadb",
"redis",
"romm",
"copyparty",
"jellyfin",
"calibre-web",
"prowlarr",
"qbittorrent",
"radarr",
"sonarr",
"lidarr",
"readarr",
"frank-default-large",
"frank-default-npu",
"frank-default-small",
)
DANKCLAW_SERVICES = {
"gluetun",
"searxng-valkey",
"searxng",
"pihole",
"open-webui",
"gitea-postgres",
"gitea-server",
"gitea-caddy",
"comfyui",
"hermes",
"hermes-dashboard",
"prometheus",
"grafana",
"loki",
"alloy",
"node-exporter",
"authentik-postgres",
"authentik-server",
"authentik-worker",
"caddy",
"homepage",
"frank-default-large",
"frank-default-npu",
}
DANKSERVER_SERVICES = {
"mariadb",
"redis",
"romm",
"copyparty",
"jellyfin",
"calibre-web",
"prowlarr",
"qbittorrent",
"radarr",
"sonarr",
"lidarr",
"readarr",
"homepage",
"frank-default-small",
}
DANKCLAW_COMPOSE_SERVICES = {
"comfyui",
"hermes",
"hermes-dashboard",
"prometheus",
"grafana",
"loki",
"alloy",
"node-exporter",
"authentik-postgres",
"authentik-server",
"authentik-worker",
"caddy",
}
DANKSERVER_COMPOSE_SERVICES = {
"mariadb",
"redis",
"romm",
"copyparty",
"jellyfin",
"calibre-web",
"prowlarr",
"qbittorrent",
"radarr",
"sonarr",
"lidarr",
"readarr",
}
def _add(services: set[str], *names: str) -> None:
services.update(names)
def target_services(target: str) -> set[str]:
if target == "dankclaw":
return DANKCLAW_SERVICES
if target == "dankserver":
return DANKSERVER_SERVICES
raise ValueError(f"unknown target: {target}")
def _compose_services(target: str) -> set[str]:
return DANKCLAW_COMPOSE_SERVICES if target == "dankclaw" else DANKSERVER_COMPOSE_SERVICES
def _git_show(repo: str, revision: str, path: str) -> str | None:
result = subprocess.run(
["git", "-C", repo, "show", f"{revision}:{path}"],
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
check=False,
)
return result.stdout if result.returncode == 0 else None
def _yaml_document(contents: str | None) -> dict[str, Any]:
if contents is None:
return {}
try:
data = yaml.safe_load(contents) or {}
except yaml.YAMLError:
return {}
return data if isinstance(data, dict) else {}
def _yaml_services(contents: str | None) -> dict[str, Any]:
services = _yaml_document(contents).get("services", {})
return services if isinstance(services, dict) else {}
def compose_changes(repo: str, target: str, base: str, head: str) -> set[str]:
"""Return only compose services whose YAML definitions or dependencies changed."""
path = f"{target}/compose.yaml"
old_doc = _yaml_document(_git_show(repo, base, path))
new_doc = _yaml_document(_git_show(repo, head, path))
old = old_doc.get("services", {}) if isinstance(old_doc.get("services", {}), dict) else {}
new = new_doc.get("services", {}) if isinstance(new_doc.get("services", {}), dict) else {}
names = set(old) | set(new)
allowed = _compose_services(target)
# An unavailable or malformed revision is safer as a broad compose plan
# than as a silent no-op. It still cannot include LLM units.
if (old_doc and not new_doc) or (new_doc and not old_doc and names):
return allowed
# Top-level networks/configuration can affect every compose service even
# when no individual service mapping changed. Ignore the services key while
# comparing the rest of the Compose document.
old_globals = {key: value for key, value in old_doc.items() if key != "services"}
new_globals = {key: value for key, value in new_doc.items() if key != "services"}
if old_globals != new_globals:
return allowed
return {name for name in names if name in allowed and old.get(name) != new.get(name)}
def services_for_path(path: str, target: str) -> set[str]:
"""Return services affected by one repository-relative path."""
path = path.replace("\\", "/")
p = PurePosixPath(path)
parts = p.parts
services: set[str] = set()
if not parts:
return services
# Non-runtime material is deliberately a no-op.
if parts[0] in {"docs", "tests", "old", ".githooks"}:
return services
if path in {"README.md", "AGENTS.md"} or parts[0] == ".obsidian":
return services
if parts[0] == "shared":
if len(parts) < 2:
return services
group = parts[1]
if group == "searxng":
_add(services, "searxng")
elif group == "prometheus":
_add(services, "prometheus")
elif group == "loki":
_add(services, "loki")
elif group == "alloy":
_add(services, "alloy")
elif group == "homepage":
# Homepage is a Dankclaw-owned deployment. Shared homepage files
# are intentionally not copied during a Dankserver-only run.
if target == "dankclaw":
_add(services, "homepage")
return services
if parts[0] == "dankclaw":
if len(parts) >= 2 and parts[1] == "compose.yaml":
# Revision-aware callers replace this broad fallback with the
# service-level YAML comparison in compose_changes().
_add(services, *DANKCLAW_COMPOSE_SERVICES)
return services
if len(parts) >= 2 and parts[1] == "Caddyfile":
_add(services, "caddy")
return services
if len(parts) >= 2 and parts[1] == "Containerfile.comfyui":
_add(services, "comfyui")
return services
if len(parts) >= 2 and parts[1] in {"caddy", "comfyui", "loki", "alloy", "hermes"}:
_add(services, parts[1])
return services
if len(parts) >= 2 and parts[1] == "searxng":
if len(parts) >= 4 and parts[2] == "quadlet":
name = parts[3]
if name == "gluetun.container":
# SearXNG shares Gluetun's network namespace. A Gluetun
# recreation requires both units to be restarted.
_add(services, "gluetun", "searxng")
elif name == "searxng.container":
_add(services, "searxng")
elif name == "searxng-valkey.container":
_add(services, "searxng-valkey", "searxng")
elif name == "searxng.network":
_add(services, "gluetun", "searxng-valkey", "searxng")
return services
if len(parts) >= 2 and parts[1] == "pihole":
_add(services, "pihole")
return services
if len(parts) >= 2 and parts[1] == "open-webui":
_add(services, "open-webui")
return services
if len(parts) >= 2 and parts[1] == "gitea":
if len(parts) >= 4 and parts[2] == "quadlet":
name = parts[3]
if name == "gitea-postgres.container":
_add(services, "gitea-postgres")
elif name == "gitea-server.container":
_add(services, "gitea-server")
elif name == "gitea-caddy.container":
_add(services, "gitea-caddy")
elif name == "gitea.network":
_add(services, "gitea-postgres", "gitea-server", "gitea-caddy")
return services
return services
if parts[0] == "dankserver":
if len(parts) >= 2 and parts[1] == "compose.yaml":
_add(services, *DANKSERVER_COMPOSE_SERVICES)
return services
if parts[0] == "scripts":
name = p.name
if name in {"frank-default-large.env", "frank-default-large.service", "serve-frank-large.sh"}:
_add(services, "frank-default-large")
elif name in {"frank-default-npu.env", "frank-default-npu.service", "serve-frank-default-npu.sh"}:
_add(services, "frank-default-npu")
elif name in {"frank-default-small.env", "frank-default-small.service", "serve-frank-small.sh"}:
_add(services, "frank-default-small")
elif name == "chat-template.jinja":
# Only the selected target's LLM consumes this shared template.
_add(services, "frank-default-large" if target == "dankclaw" else "frank-default-small")
elif name in {"homepage-status.py", "homepage-status.service", "homepage-status.timer"}:
_add(services, "homepage")
# Deployment/test/benchmark scripts and other repo tooling do not
# restart a runtime service.
return services
# Examples and env templates are not deployed runtime state.
if p.name in {".env.example", "env.example"}:
return services
return services
def changed_paths(repo: str, base: str, head: str) -> list[str]:
result = subprocess.run(
["git", "-C", repo, "diff", "--name-only", "--diff-filter=ACDMRTUXB", base, head],
text=True,
stdout=subprocess.PIPE,
check=True,
)
return [line for line in result.stdout.splitlines() if line]
def plan(paths: list[str], target: str, repo: str | None = None,
base: str | None = None, head: str | None = None) -> list[str]:
services: set[str] = set()
for path in paths:
if path == f"{target}/compose.yaml" and repo and base and head:
services.update(compose_changes(repo, target, base, head))
else:
services.update(services_for_path(path, target))
allowed = target_services(target)
return [name for name in SERVICE_ORDER if name in services and name in allowed]
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--target", choices=("dankclaw", "dankserver"), required=True)
parser.add_argument("--paths", nargs="+")
parser.add_argument("--repo", help="repository path for revision-aware planning")
parser.add_argument("--base", help="base Git revision")
parser.add_argument("--head", help="head Git revision")
args = parser.parse_args()
if not args.paths and not (args.repo and args.base and args.head):
parser.error("provide --paths or all of --repo, --base, and --head")
paths = args.paths or changed_paths(args.repo, args.base, args.head)
print("\n".join(plan(paths, args.target, args.repo, args.base, args.head)))
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/deploy-remote.sh
Executable
+474
View File
@@ -0,0 +1,474 @@
#!/usr/bin/env bash
# deploy-remote.sh — apply an explicit service plan on one target host.
#
# This script runs on the target checkout. It deliberately does not infer a
# plan, pull Git, or restart an unrelated service. deploy.sh computes the plan
# from the pushed revision and passes the selected service names here.
# Per-service revisions live in /var/lib/serverAppConfigs/deploy-state (or
# DEPLOY_STATE_DIR), outside the checkout, so an unselected change remains
# pending until its service is explicitly deployed.
set -euo pipefail
TARGET="${1:-}"
REVISION="${2:-}"
shift 2 || true
FROM_REVISION=""
PENDING=0
SERVICES=()
while [[ "$#" -gt 0 ]]; do
case "$1" in
--from-revision)
[[ "$#" -ge 2 ]] || { echo "--from-revision requires a revision" >&2; exit 2; }
FROM_REVISION="$2"
shift 2
;;
--pending)
PENDING=1
shift
;;
--help|-h)
usage
exit 0
;;
*)
SERVICES+=("$1")
shift
;;
esac
done
REMOTE_DIR="${REMOTE_DIR:-/home/dank/serverAppConfigs}"
STATE_DIR="${DEPLOY_STATE_DIR:-${XDG_STATE_HOME:-$HOME/.local/state}/serverAppConfigs/deploy-state}"
run_privileged() {
if [[ -n "${FRANK_SUDO_PW:-}" ]]; then
printf '%s\n' "$FRANK_SUDO_PW" | sudo -S -- "$@"
else
sudo -- "$@"
fi
}
usage() {
cat <<'EOF'
Usage: deploy-remote.sh <dankclaw|dankserver> <git-revision> [--pending] [service]...
The service list is explicit. An empty list is a successful no-op.
EOF
}
if [[ "$TARGET" != dankclaw && "$TARGET" != dankserver ]]; then
usage >&2
exit 2
fi
if [[ -z "$REVISION" ]]; then
usage >&2
exit 2
fi
if [[ "$REVISION" == *[!0-9a-fA-F]* ]]; then
echo "invalid Git revision: $REVISION" >&2
exit 2
fi
if ! git -C "$REMOTE_DIR" cat-file -e "$REVISION^{commit}" 2>/dev/null; then
echo "revision is not a commit in the target checkout: $REVISION" >&2
exit 2
fi
cd "$REMOTE_DIR"
# Keep the state directory outside Git. The file is written only after a
# handler completes successfully; an interrupted handler remains pending.
mkdir -p "$STATE_DIR"
service_is_allowed() {
local service="$1"
case "$TARGET:$service" in
dankclaw:gluetun|dankclaw:searxng-valkey|dankclaw:searxng|dankclaw:pihole|\
dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
dankclaw:gitea-caddy|dankclaw:comfyui|dankclaw:hermes|\
dankclaw:hermes-dashboard|dankclaw:prometheus|dankclaw:grafana|\
dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|\
dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:caddy|dankclaw:homepage|\
dankclaw:frank-default-large|dankclaw:frank-default-npu|\
dankserver:mariadb|dankserver:redis|dankserver:romm|\
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|\
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
dankserver:frank-default-small)
return 0
;;
*)
return 1
;;
esac
}
mark_status() {
local service="$1" status="$2" tmp
tmp="$(mktemp "$STATE_DIR/.${service}.XXXXXX")"
printf 'revision=%s\nstatus=%s\n' "$REVISION" "$status" > "$tmp"
mv -f -- "$tmp" "$STATE_DIR/$service"
}
mark_deployed() {
mark_status "$1" deployed
}
state_revision() {
local service="$1" line value
if [[ -r "$STATE_DIR/$service" ]]; then
while IFS= read -r line; do
case "$line" in
revision=*) value="${line#revision=}" ;;
esac
done < "$STATE_DIR/$service"
printf '%s' "${value:-}"
fi
}
state_status() {
local service="$1" line value
if [[ -r "$STATE_DIR/$service" ]]; then
while IFS= read -r line; do
case "$line" in
status=*) value="${line#status=}" ;;
esac
done < "$STATE_DIR/$service"
printf '%s' "${value:-pending}"
fi
}
add_changed_services() {
local base="$1"
local service
[[ -n "$base" ]] || return 0
while IFS= read -r service; do
[[ -n "$service" ]] || continue
SERVICES+=("$service")
done < <(python3 scripts/deploy-plan.py --target "$TARGET" --repo "$REMOTE_DIR" \
--base "$base" --head "$REVISION")
}
add_pending_services() {
local service recorded changed
for state_file in "$STATE_DIR"/*; do
[[ -f "$state_file" ]] || continue
service="${state_file##*/}"
recorded="$(state_revision "$service")"
[[ -n "$recorded" && "$recorded" != "$REVISION" ]] || continue
[[ "$(state_status "$service")" != deployed ]] || continue
if [[ "$TARGET" == dankclaw && "$service" == frank-default-small ]]; then continue; fi
if [[ "$TARGET" == dankserver && "$service" == frank-default-large ]]; then continue; fi
if ! changed="$(python3 scripts/deploy-plan.py --target "$TARGET" \
--repo "$REMOTE_DIR" --base "$recorded" --head "$REVISION")"; then
echo "WARN: unable to compare state for $service; retaining it as pending" >&2
SERVICES+=("$service")
continue
fi
if grep -Fxq "$service" <<< "$changed"; then
SERVICES+=("$service")
fi
done
}
copy_shared_config() {
run_privileged mkdir -p /opt/searxng/config /opt/prometheus/config /opt/loki/config /opt/alloy/config
run_privileged cp shared/searxng/settings.yml /opt/searxng/config/settings.yml
run_privileged cp shared/searxng/limiter.toml /opt/searxng/config/limiter.toml
run_privileged cp shared/prometheus/prometheus.yml /opt/prometheus/config/prometheus.yml
run_privileged cp shared/loki/loki-config.yaml /opt/loki/config/loki-config.yaml
run_privileged cp shared/alloy/config.alloy /opt/alloy/config/config.alloy
}
render_caddyfile() {
local hash
hash="$(grep -E '^CADDY_BASICAUTH_HASH=' .env 2>/dev/null | tail -1 | cut -d= -f2- || true)"
if [[ -z "$hash" ]]; then
echo "WARN: CADDY_BASICAUTH_HASH is not set; refusing to overwrite rendered Caddyfile" >&2
return 1
fi
run_privileged mkdir -p /home/dank/Containers/caddy
sed -e "s|REPLACE_WITH_HASH|$hash|g" dankclaw/Caddyfile > /tmp/serverAppConfigs.Caddyfile
run_privileged cp /tmp/serverAppConfigs.Caddyfile /home/dank/Containers/caddy/Caddyfile
}
deploy_homepage() {
run_privileged mkdir -p /srv/homepage
run_privileged cp shared/homepage/index.html shared/homepage/admin.html shared/homepage/app.js \
shared/homepage/style.css shared/homepage/services.json /srv/homepage/
if [[ -d shared/homepage/fonts ]]; then
run_privileged cp -r shared/homepage/fonts /srv/homepage/
fi
local deployed_at
deployed_at="$(git log -1 --format=%cI)"
printf 'window.DEPLOYED_AT = "%s";\n' "$deployed_at" > /tmp/deployed.js
run_privileged cp /tmp/deployed.js /srv/homepage/deployed.js
run_privileged chmod -R a+rX /srv/homepage
run_privileged cp scripts/homepage-status.py /usr/local/bin/homepage-status.py
run_privileged cp scripts/homepage-status.service scripts/homepage-status.timer /etc/systemd/system/
run_privileged systemctl daemon-reload
run_privileged systemctl enable homepage-status.timer
run_privileged systemctl restart homepage-status.timer
run_privileged python3 /usr/local/bin/homepage-status.py /srv/homepage/services.json /srv/homepage/status.json
}
deploy_caddy() {
copy_shared_config
render_caddyfile
deploy_homepage
local cfhash oldhash image
cfhash="$(sha256sum dankclaw/caddy/Containerfile | cut -d' ' -f1)"
image=franksdank/caddy-porkbun
oldhash="$(podman image inspect "$image:latest" --format '{{index .Config.Labels "cfhash"}}' 2>/dev/null || echo none)"
if [[ "$oldhash" != "$cfhash" ]]; then
podman build --no-cache --label "cfhash=$cfhash" -t "$image:latest" \
-f dankclaw/caddy/Containerfile .
fi
# The canonical reverse proxy may be gitea-caddy on a host with the old
# split deployment. Never stop or replace it implicitly. Refuse the Caddy
# deployment until the operator has explicitly migrated port ownership.
if podman container exists gitea-caddy && ! podman container exists caddy; then
echo "FAIL: gitea-caddy owns the reverse-proxy deployment; refusing to replace it with caddy." >&2
return 1
fi
if podman container exists gitea-caddy && podman container exists caddy; then
local gitea_state caddy_state
gitea_state="$(podman inspect gitea-caddy --format '{{.State.Status}}' 2>/dev/null || true)"
caddy_state="$(podman inspect caddy --format '{{.State.Status}}' 2>/dev/null || true)"
if [[ "$gitea_state" == running && "$caddy_state" != running ]]; then
echo "FAIL: gitea-caddy currently owns ports 80/443; refusing to start caddy." >&2
return 1
fi
fi
if podman container exists caddy; then
podman-compose -f dankclaw/compose.yaml up -d --no-deps caddy
local reloaded=0
for _ in 1 2 3 4 5; do
if podman exec caddy caddy reload --config /etc/caddy/Caddyfile \
--adapter caddyfile >/dev/null 2>&1; then
reloaded=1
echo "Caddy reloaded."
break
fi
sleep 3
done
if [[ "$reloaded" -ne 1 ]]; then
echo "Caddy reload failed; restarting only caddy."
podman restart caddy
fi
else
echo "WARN: caddy container is absent; no proxy was stopped or replaced."
fi
}
deploy_compose_service() {
local service="$1"
if [[ "$TARGET" == dankserver ]]; then
# Dankserver's compose file is the source-of-truth inventory, but its
# production containers are individually systemd-managed/bare. Do not
# use compose up here: it can create a second container with a wrong
# name and can interfere with a running workload.
echo "WARN: $service is recorded in dankserver/compose.yaml; production container management is host-specific and no restart was attempted."
return 0
fi
case "$service" in
comfyui|hermes|hermes-dashboard|prometheus|grafana|loki|alloy|node-exporter|authentik-postgres|authentik-server|authentik-worker)
case "$service" in
comfyui|loki|alloy)
build_compose_image "$service"
;;
esac
podman-compose -f dankclaw/compose.yaml up -d --no-deps "$service"
;;
*)
echo "unknown compose service: $service" >&2
return 2
;;
esac
}
build_compose_image() {
local service="$1" image cfhash oldhash containerfile
case "$service" in
comfyui)
image=comfyui-rocm:gfx1151
containerfile=dankclaw/Containerfile.comfyui
;;
loki|alloy)
image="franksdank/$service:latest"
containerfile="dankclaw/$service/Containerfile"
;;
*)
return 0
;;
esac
cfhash="$(sha256sum "$containerfile" | cut -d' ' -f1)"
oldhash="$(podman image inspect "$image" --format '{{index .Config.Labels "cfhash"}}' 2>/dev/null || echo none)"
if [[ "$oldhash" != "$cfhash" ]]; then
podman build --no-cache --label "cfhash=$cfhash" -t "$image" -f "$containerfile" .
fi
}
quadlet_unit_dir="$HOME/.config/containers/systemd"
install_quadlet() {
local service="$1"
local source_dir="$2"
local unit="$3"
mkdir -p "$quadlet_unit_dir"
cp "$source_dir/$unit.container" "$quadlet_unit_dir/"
if [[ -f "$source_dir/$unit.network" ]]; then
cp "$source_dir/$unit.network" "$quadlet_unit_dir/"
fi
systemctl --user daemon-reload
}
deploy_searxng() {
copy_shared_config
install_quadlet searxng dankclaw/searxng/quadlet searxng
install_quadlet searxng-valkey dankclaw/searxng/quadlet searxng-valkey
install_quadlet gluetun dankclaw/searxng/quadlet gluetun
# The network is shared by all three units; install it even when only the
# SearXNG service was selected. These copies are user-owned Quadlet state.
cp dankclaw/searxng/quadlet/searxng.network "$quadlet_unit_dir/"
systemctl --user daemon-reload
systemctl --user start searxng-network.service
systemctl --user restart gluetun.service searxng-valkey.service searxng.service
}
deploy_gluetun() {
install_quadlet gluetun dankclaw/searxng/quadlet gluetun
cp dankclaw/searxng/quadlet/searxng.network "$quadlet_unit_dir/"
systemctl --user daemon-reload
systemctl --user restart gluetun.service searxng.service
}
deploy_searxng_valkey() {
install_quadlet searxng-valkey dankclaw/searxng/quadlet searxng-valkey
cp dankclaw/searxng/quadlet/searxng.network "$quadlet_unit_dir/"
systemctl --user daemon-reload
systemctl --user restart searxng-valkey.service searxng.service
}
deploy_open_webui() {
install_quadlet open-webui dankclaw/open-webui/quadlet open-webui
mkdir -p /home/dank/Containers/open-webui/data
systemctl --user daemon-reload
systemctl --user restart open-webui.service
}
deploy_pihole() {
mkdir -p /home/dank/Containers/pihole/etc-pihole /home/dank/Containers/pihole/etc-dockers
if [[ ! -f /home/dank/Containers/pihole/env ]]; then
cp dankclaw/pihole/env.example /home/dank/Containers/pihole/env
echo "WARN: created Pi-hole env from the example; set WEBPASSWORD before using its UI" >&2
fi
cp dankclaw/pihole/addn-hosts /home/dank/Containers/pihole/addn-hosts
mkdir -p "$quadlet_unit_dir"
cp dankclaw/pihole/quadlet/pihole.container "$quadlet_unit_dir/"
systemctl --user daemon-reload
systemctl --user restart pihole.service
}
deploy_gitea_unit() {
local service="$1"
local unit="$2"
mkdir -p "$quadlet_unit_dir"
cp "dankclaw/gitea/quadlet/$unit.container" "$quadlet_unit_dir/"
[[ -f dankclaw/gitea/quadlet/gitea.network ]] && cp dankclaw/gitea/quadlet/gitea.network "$quadlet_unit_dir/"
systemctl --user daemon-reload
systemctl --user restart "$service.service"
}
relink_unit() {
local service="$1"
run_privileged ln -sfn "$REMOTE_DIR/scripts/$service.service" "/etc/systemd/system/$service.service"
}
deploy_llm_large() {
relink_unit frank-default-large
# NPU is intentionally not restarted here. It is a separate, retired
# endpoint and its unit is only relinked when explicitly selected.
run_privileged systemctl daemon-reload
run_privileged systemctl enable frank-default-large
run_privileged systemctl restart frank-default-large
wait_for_health frank-default-large http://127.0.0.1:7770/health 1200
}
deploy_llm_small() {
relink_unit frank-default-small
run_privileged systemctl daemon-reload
run_privileged systemctl enable frank-default-small
run_privileged systemctl restart frank-default-small
wait_for_health frank-default-small http://127.0.0.1:7770/health 1200
}
deploy_llm_npu() {
relink_unit frank-default-npu
run_privileged systemctl daemon-reload
echo "Relinked frank-default-npu; it was not restarted."
}
wait_for_health() {
local service="$1" url="$2" limit="$3"
local i
for ((i=1; i<=limit; i++)); do
if curl -fsS "$url" >/dev/null 2>&1; then
echo "$service healthy after approximately $((i * 2))s"
return 0
fi
sleep 2
done
echo "FAIL: $service did not become healthy in $((limit * 2))s" >&2
return 1
}
deploy_service() {
local service="$1"
case "$service" in
caddy) deploy_caddy ;;
homepage) deploy_homepage ;;
searxng) deploy_searxng ;;
gluetun) deploy_gluetun ;;
searxng-valkey) deploy_searxng_valkey ;;
open-webui) deploy_open_webui ;;
pihole) deploy_pihole ;;
gitea-postgres) deploy_gitea_unit gitea-postgres gitea-postgres ;;
gitea-server) deploy_gitea_unit gitea-server gitea-server ;;
gitea-caddy) deploy_gitea_unit gitea-caddy gitea-caddy ;;
frank-default-large) deploy_llm_large ;;
frank-default-small) deploy_llm_small ;;
frank-default-npu) deploy_llm_npu ;;
*) deploy_compose_service "$service" ;;
esac
}
if [[ "$PENDING" -eq 1 ]]; then
add_changed_services "$FROM_REVISION"
add_pending_services
fi
# Remove duplicate service names while preserving caller/planner order.
UNIQUE_SERVICES=()
declare -A SEEN_SERVICES=()
for service in "${SERVICES[@]}"; do
[[ -n "$service" ]] || continue
if [[ -z "${SEEN_SERVICES[$service]:-}" ]]; then
UNIQUE_SERVICES+=("$service")
SEEN_SERVICES[$service]=1
fi
done
SERVICES=("${UNIQUE_SERVICES[@]}")
if [[ "${#SERVICES[@]}" -eq 0 ]]; then
echo "No services selected; nothing to deploy."
exit 0
fi
for service in "${SERVICES[@]}"; do
[[ "$service" =~ ^[a-z0-9][a-z0-9-]*$ ]] || { echo "invalid service: $service" >&2; exit 2; }
service_is_allowed "$service" || { echo "service $service is not deployed on $TARGET" >&2; exit 2; }
echo "=== Deploying $service at $REVISION ==="
mark_status "$service" pending
deploy_service "$service"
mark_deployed "$service"
echo " STATE[$service] = $REVISION (status=deployed)"
done
scripts/deploy-searxng-quadlet.sh
+12 -93
View File
@@ -1,97 +1,16 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# deploy-searxng-quadlet.sh — Install and (re)start the SearXNG quadlet stack. #!/usr/bin/env bash
# # Compatibility wrapper for the service-scoped deployment dispatcher.
# The gluetun + searxng + valkey trio is boot-persistent via Quadlet, not # Use deploy.sh for Git push/pull and planning; this name remains for operators
# podman-compose (see AGENTS.md). This script: # and runbooks that historically deployed the Quadlet trio directly.
# 1. Pushes main to origin
# 2. Pulls the repo on the target
# 3. Copies the quadlet units into ~/.config/containers/systemd/
# 4. daemon-reloads, enables the units for boot, and starts them in order
#
# It is idempotent: it stops any running units and removes any containers with
# the trio's names (compose leftovers or the previous quadlet run) before
# reinstalling and starting, so it is safe to run on first migration and on
# every later config change.
#
# Usage:
# ./scripts/deploy-searxng-quadlet.sh <target> [--dry-run]
# ./scripts/deploy-searxng-quadlet.sh dankclaw
set -euo pipefail set -euo pipefail
REMOTE_USER="dank"
REMOTE_HOST="dankclaw"
REMOTE_DIR="/home/dank/serverAppConfigs"
QDIR="dankclaw/searxng/quadlet"
UNITS="searxng-network.service gluetun.service searxng-valkey.service searxng.service"
TARGET="${1:-}" TARGET="${1:-}"
DRY_RUN=0 shift || true
if [ "${2:-}" = "--dry-run" ]; then case "$TARGET" in
DRY_RUN=1 dankclaw) exec "$(dirname "$0")/deploy.sh" dankclaw --service searxng "$@" ;;
fi *)
echo "This stack runs only on dankclaw (got: ${TARGET:-<missing>})" >&2
if [ -z "${TARGET:-}" ]; then exit 2
echo "Usage: $0 <target> [--dry-run]" ;;
exit 1 esac
fi
if [ "${TARGET}" != "dankclaw" ]; then
echo "This stack runs only on dankclaw (got: ${TARGET})"
exit 1
fi
# ---------------------------------------------------------------------------
# Local pre-flight
# ---------------------------------------------------------------------------
cd "$(dirname "$0")/.."
if ! git diff --quiet HEAD; then
echo "FAIL: Local working tree has uncommitted changes. Commit first."
exit 1
fi
if [ "$DRY_RUN" = "1" ]; then
echo "[dry-run] Would push main, pull on ${REMOTE_HOST}, install quadlet units, and start: ${UNITS}"
exit 0
fi
echo "=== Deploying SearXNG quadlet stack to ${REMOTE_HOST} ==="
echo "[1/3] Pushing local main to origin..."
git push origin main
REMOTE="${REMOTE_USER}@${REMOTE_HOST}"
echo "[2/3] Installing and starting the quadlet units on ${REMOTE_HOST}..."
ssh "$REMOTE" bash -s <<REMOTE_EOF
set -euo pipefail
cd '${REMOTE_DIR}'
echo ' Pulling repo...'
git pull origin main
echo ' Stopping any running units (idempotent)...'
systemctl --user stop searxng.service gluetun.service searxng-valkey.service searxng-network.service 2>/dev/null || true
echo ' Removing any containers with the trio names (compose leftovers or prior run)...'
podman rm -f searxng searxng-valkey gluetun 2>/dev/null || true
echo ' Copying quadlet units into ~/.config/containers/systemd/ ...'
mkdir -p ~/.config/containers/systemd
cp ${QDIR}/*.container ${QDIR}/*.network ~/.config/containers/systemd/
echo ' daemon-reload (generates the units)...'
# Quadlet units are generator-managed: systemctl enable fails on them ("transient
# or generated"). The [Install] WantedBy=default.target in each .container file is
# itself the enablement (is-enabled reports "generated"), and with user linger on
# they start at boot. The .network unit is pulled in by the containers that set
# Wants=/After=searxng-network.service. So: daemon-reload, then start.
systemctl --user daemon-reload
echo ' Starting in dependency order...'
systemctl --user start searxng-network.service
systemctl --user start gluetun.service searxng-valkey.service
systemctl --user start searxng.service
REMOTE_EOF
echo "[3/3] Verifying unit state..."
ssh "$REMOTE" 'systemctl --user --no-pager status searxng-network.service gluetun.service searxng-valkey.service searxng.service 2>&1 | grep -E "●|Active:"' || true
echo "=== SearXNG quadlet stack deployed to ${REMOTE_HOST} ==="
scripts/deploy.sh
+199 -289
View File
@@ -1,304 +1,214 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# deploy.sh — Deploy the serverAppConfigs repo to a target and restart its # deploy.sh — push main, pull one target, and deploy only selected services.
# podman stack and LLM fleet services.
# #
# Implements the git deployment workflow from AGENTS.md. Assumes local # Planning is path-aware and happens before the push. The target-side runner
# changes are committed and pushed, then: # then applies only those services, with per-service deployment revisions kept
# 1. Pushes main to origin # outside the Git checkout. A docs-only commit is a no-op. An LLM service is
# 2. Pulls the repo on the target # restarted only when its own files (or the shared chat template) are selected.
# 3. Copies shared configs into place and restarts the podman stack
# (dankclaw only; the dankserver media stack runs as bare containers)
# 4. Relinks the LLM systemd units and restarts the active LLM service
# 5. Waits for the Caddy homepage and the LLM health endpoint
#
# Sudo: the fleet hosts allow passwordless sudo for the deploy user. For
# headless runs on a host without that, set FRANK_SUDO_PW so remote sudo
# reads the password from stdin (`sudo -S`). The password is never stored
# in this repo.
# #
# Usage: # Usage:
# ./scripts/deploy.sh <target> [--dry-run] # ./scripts/deploy.sh <dankclaw|dankserver> [--service NAME]... [--dry-run]
# #
# Targets: # Without --service, the planner selects services changed since origin/main and
# dankclaw - AI/search/dev/observability stack + frank-default-large # the remote runner also applies pending service changes from its state store.
# (CIRU ROCmFPX llama.cpp, Qwen3.8-27B, port 7770) # --service intentionally restricts the run to the named services; it is useful
# dankserver - media/library stack (bare containers) + frank-default-small # for a safe, explicit retry after another service failed.
# (upstream llama.cpp, Qwen3.6-35B-A3B, port 7770)
#
# Example:
# ./scripts/deploy.sh dankclaw
# ./scripts/deploy.sh dankserver --dry-run
set -euo pipefail set -euo pipefail
# --------------------------------------------------------------------------- SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
# Target configuration REPO_ROOT="$(cd -- "$SCRIPT_DIR/.." && pwd)"
# --------------------------------------------------------------------------- PLANNER="$SCRIPT_DIR/deploy-plan.py"
declare -A REMOTE_USER REMOTE_HOST REMOTE_DIR COMPOSE HEALTH_URL COMPOSE_UP COPY_SHARED REMOTE_USER="dank"
declare -A LLM_SERVICES LLM_UNITS LLM_HEALTH LLM_WAIT REMOTE_DIR="/home/dank/serverAppConfigs"
# Caddy is consolidated on dankclaw: it fronts both hosts' services. The usage() {
# dankserver media stack runs as bare containers (not this repo's compose), cat <<'EOF'
# so its stack deploy is pull-only; health is checked against the shared Caddy. Usage: deploy.sh <dankclaw|dankserver> [options]
REMOTE_USER[dankclaw]=dank
REMOTE_HOST[dankclaw]=dankclaw
REMOTE_DIR[dankclaw]="/home/dank/serverAppConfigs"
COMPOSE[dankclaw]="dankclaw/compose.yaml"
HEALTH_URL[dankclaw]="http://127.0.0.1/"
COMPOSE_UP[dankclaw]=1
COPY_SHARED[dankclaw]=1
# LLM: frank-default-large is the active service (restart + health wait).
# frank-default-npu is relinked only (not enabled on the host); its unit
# picks up the new paths at its next manual start.
LLM_SERVICES[dankclaw]="frank-default-large"
LLM_UNITS[dankclaw]="frank-default-large frank-default-npu"
LLM_HEALTH[dankclaw]="http://127.0.0.1:7770/health"
# CIRU llama.cpp (ROCm 7.15) loads the v3 five-file payload (GGUF + FFN +
# GDN + Output-K8 .pfs + mmproj, ~36.7 GiB) + MTP draft/Output-K8 proxy and
# initializes PromptForge views; allow 20 minutes.
LLM_WAIT[dankclaw]=600
REMOTE_USER[dankserver]=dank Options:
REMOTE_HOST[dankserver]=dankserver --service NAME deploy only NAME (repeatable; bypasses automatic planning)
REMOTE_DIR[dankserver]="/home/dank/serverAppConfigs" --dry-run show the plan without pushing or connecting to a target
COMPOSE[dankserver]="dankserver/compose.yaml" -h, --help show this help
HEALTH_URL[dankserver]="http://192.168.0.143/"
COMPOSE_UP[dankserver]=0 Examples:
COPY_SHARED[dankserver]=0 ./scripts/deploy.sh dankclaw
# LLM: frank-default-small is the active service (restart + health wait). ./scripts/deploy.sh dankclaw --service searxng
LLM_SERVICES[dankserver]="frank-default-small" ./scripts/deploy.sh dankserver --service frank-default-small
LLM_UNITS[dankserver]="frank-default-small" ./scripts/deploy.sh dankclaw --dry-run
LLM_HEALTH[dankserver]="http://127.0.0.1:7770/health" EOF
# Cold load: --no-mmap reads the 30.6 GB Q6_K_P through RAM (MoE experts on }
# CPU); allow 20 minutes.
LLM_WAIT[dankserver]=600
TARGET="${1:-}" TARGET="${1:-}"
if [[ "$TARGET" == "-h" || "$TARGET" == "--help" ]]; then
usage
exit 0
fi
if [[ -z "$TARGET" ]]; then
usage >&2
exit 2
fi
shift
case "$TARGET" in
dankclaw) REMOTE_HOST="dankclaw" ;;
dankserver) REMOTE_HOST="dankserver" ;;
*)
echo "Unknown target: $TARGET" >&2
usage >&2
exit 2
;;
esac
DRY_RUN=0 DRY_RUN=0
if [ "${2:-}" = "--dry-run" ]; then EXPLICIT_SERVICES=()
DRY_RUN=1 while [[ "$#" -gt 0 ]]; do
fi case "$1" in
--dry-run)
# --------------------------------------------------------------------------- DRY_RUN=1
# Validation shift
# --------------------------------------------------------------------------- ;;
if [ -z "${TARGET:-}" ]; then --service)
echo "Usage: $0 <dankclaw|dankserver> [--dry-run]" [[ "$#" -ge 2 ]] || { echo "--service requires a name" >&2; exit 2; }
exit 1 EXPLICIT_SERVICES+=("$2")
fi shift 2
if [ -z "${REMOTE_HOST[$TARGET]:-}" ]; then ;;
echo "Unknown target: $TARGET" -h|--help)
exit 1 usage
fi
# ---------------------------------------------------------------------------
# Sudo wrapper (headless-friendly)
# ---------------------------------------------------------------------------
SUDO="sudo"
if [ -n "${FRANK_SUDO_PW:-}" ]; then
SUDO="printf '%s\n' '$FRANK_SUDO_PW' | sudo -S"
fi
# ---------------------------------------------------------------------------
# Local checks (pre-flight)
# ---------------------------------------------------------------------------
cd "$(dirname "$0")/.."
if ! git diff --quiet HEAD; then
echo "FAIL: Local working tree has uncommitted changes."
echo "Commit or stash them before deploying."
exit 1
fi
if [ "$(git rev-parse --abbrev-ref HEAD)" != "main" ]; then
echo "WARN: Not on 'main'; deploying a non-main branch."
fi
echo "=== Deploying to ${TARGET} ==="
echo " host: ${REMOTE_HOST[$TARGET]}"
echo " dir: ${REMOTE_DIR[$TARGET]}"
echo " compose: ${COMPOSE[$TARGET]}"
echo " llm svc: ${LLM_SERVICES[$TARGET]}"
echo " llm health: ${LLM_HEALTH[$TARGET]}"
echo " homepage: ${HEALTH_URL[$TARGET]}"
echo ""
# ---------------------------------------------------------------------------
# Push (unless dry-run)
# ---------------------------------------------------------------------------
if [ "$DRY_RUN" = "1" ]; then
echo "[dry-run] Would push local main to origin"
echo "[dry-run] Would SSH to ${REMOTE_HOST[$TARGET]} and git pull"
else
echo "[1/5] Pushing local main to origin..."
git push origin main
fi
REMOTE="$(
printf '%s@%s' "${REMOTE_USER[$TARGET]}" "${REMOTE_HOST[$TARGET]}"
)"
# ---------------------------------------------------------------------------
# Remote: pull + copy shared configs + restart stack + restart LLM service
# ---------------------------------------------------------------------------
REMOTE_SCRIPT="
set -euo pipefail
cd '${REMOTE_DIR[$TARGET]}'
echo '[2/5] Pulling repo on ${TARGET}...'
git pull origin main
if [ -f .env ]; then
echo ' .env present, using it'
else
echo ' WARN: no .env on target; copy the .env.example and fill it in'
fi
if [ '${COPY_SHARED[$TARGET]}' = '1' ]; then
echo '[3/5] Copying shared configs into place...'
mkdir -p /opt/searxng/config /opt/prometheus/config /opt/loki/config /opt/alloy/config
${SUDO} cp shared/searxng/settings.yml /opt/searxng/config/settings.yml
${SUDO} cp shared/searxng/limiter.toml /opt/searxng/config/limiter.toml
${SUDO} cp shared/prometheus/prometheus.yml /opt/prometheus/config/prometheus.yml
${SUDO} cp shared/loki/loki-config.yaml /opt/loki/config/loki-config.yaml
${SUDO} cp shared/alloy/config.alloy /opt/alloy/config/config.alloy
echo ' Rendering Caddyfile (basicauth hash from .env)...'
# Extract the hash with grep/cut rather than sourcing .env: sourcing a
# bcrypt value in a set -u shell dies on its dollar-sign expansion.
# '|| true' so a missing hash warns below instead of aborting under set -e.
# Only the shared family hash remains: the homepage's per-user hashes were
# retired in favor of Authentik forward_auth (the /admin/ page and the user
# page are now behind the same SSO session).
CADDY_BASICAUTH_HASH=\"\$(grep -E '^CADDY_BASICAUTH_HASH=' .env 2>/dev/null | tail -1 | cut -d= -f2- || true)\"
if [ -n \"\${CADDY_BASICAUTH_HASH}\" ]; then
# '|' delimiter: the bcrypt hash contains '/' characters.
mkdir -p /home/dank/Containers/caddy
sed -e \"s|REPLACE_WITH_HASH|\${CADDY_BASICAUTH_HASH}|g\" \\
dankclaw/Caddyfile > /home/dank/Containers/caddy/Caddyfile
else
echo ' WARN: CADDY_BASICAUTH_HASH is not set in .env; keeping the placeholder(s)'
fi
echo ' Deploying the homepage...'
${SUDO} mkdir -p /srv/homepage
${SUDO} cp shared/homepage/index.html shared/homepage/admin.html shared/homepage/app.js \\
shared/homepage/style.css shared/homepage/services.json /srv/homepage/
if [ -d shared/homepage/fonts ]; then ${SUDO} cp -r shared/homepage/fonts /srv/homepage/; fi
# The last master deployment datetime (Q43): the deployed commit's datetime.
DEPLOYED_AT=\"\$(git log -1 --format=%cI)\"
printf 'window.DEPLOYED_AT = \"%s\";\\n' \"\$DEPLOYED_AT\" > /tmp/deployed.js
${SUDO} cp /tmp/deployed.js /srv/homepage/deployed.js
${SUDO} chmod -R a+rX /srv/homepage
# The status checker: install the timer + service, run it once, enable it.
${SUDO} cp scripts/homepage-status.py /usr/local/bin/homepage-status.py
${SUDO} cp scripts/homepage-status.service scripts/homepage-status.timer /etc/systemd/system/
${SUDO} systemctl daemon-reload
${SUDO} systemctl enable homepage-status.timer
${SUDO} systemctl restart homepage-status.timer
${SUDO} python3 /usr/local/bin/homepage-status.py /srv/homepage/services.json /srv/homepage/status.json
fi
if [ '${COMPOSE_UP[$TARGET]}' = '1' ]; then
echo '[4/5] Restarting stack...'
# Locally built images: caddy + porkbun module, and loki/alloy + a static
# busybox (their upstream images lack the wget the healthchecks need).
for dir in caddy loki alloy; do
img=franksdank/\$dir
[ \"\$dir\" = caddy ] && img=franksdank/caddy-porkbun
# Rebuild when the Containerfile changed. buildah's layer cache can
# serve a stale COPY --from layer (keyed on stage name), so a rebuild
# always runs with --no-cache.
cfhash=\$(sha256sum dankclaw/\$dir/Containerfile | cut -d' ' -f1)
oldhash=\$(podman image inspect \$img:latest --format '{{index .Config.Labels \"cfhash\"}}' 2>/dev/null || echo none)
if [ \"\$oldhash\" != \"\$cfhash\" ]; then
echo \" \$img image missing or stale; building it...\"
podman build --no-cache --label cfhash=\$cfhash -t \$img:latest -f dankclaw/\$dir/Containerfile .
fi
done
if ! podman-compose -f '${COMPOSE[$TARGET]}' up -d; then
echo ' WARN: compose restart failed; continuing with LLM deployment.' >&2
fi
# A Caddyfile render is a bind-mount content change, not a container
# config change: the caddy container often survives 'up -d' and keeps
# serving its in-memory config. Reload explicitly; the file watcher
# (admin on) is the backstop if the API is not up yet. If the reload
# cannot be applied at all (e.g. the running config has the admin API
# disabled -- the fix cannot ride the API it is meant to enable),
# restart the container so it reads the file on start.
caddy_reloaded=0
for i in 1 2 3 4 5; do
if podman exec caddy caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile >/dev/null 2>&1; then
caddy_reloaded=1
echo \" Caddy reloaded.\"
break
fi
sleep 3
done
if [ \"\$caddy_reloaded\" -ne 1 ]; then
echo \" Caddy reload failed; restarting the container to load the file.\"
podman restart caddy || true
fi
else
echo '[4/5] Skipping compose restart (stack runs as bare containers on this host)'
fi
echo '[4/5] Relinking LLM units and restarting LLM services...'
for unit in ${LLM_UNITS[$TARGET]}; do
# The loop variable must expand on the remote, so it is escaped here and
# the path is unquoted (it has no spaces). Quoting the source path would
# ship a literal loop-variable token and create a dangling symlink
# (the 2026-08-22 relink bug).
${SUDO} ln -sf ${REMOTE_DIR[$TARGET]}/scripts/\${unit}.service /etc/systemd/system/\${unit}.service
done
${SUDO} systemctl daemon-reload
for svc in ${LLM_SERVICES[$TARGET]}; do
${SUDO} systemctl enable \"\${svc}\"
${SUDO} systemctl restart \"\${svc}\"
echo \" restarted \${svc}\"
done
"
if [ "$DRY_RUN" = "1" ]; then
echo "[dry-run] Remote commands would run:"
echo "$REMOTE_SCRIPT"
else
# shellcheck disable=SC2087
ssh "$REMOTE" bash -s <<REMOTE_EOF
${REMOTE_SCRIPT}
REMOTE_EOF
fi
# ---------------------------------------------------------------------------
# Health checks: homepage, then the LLM endpoint
# ---------------------------------------------------------------------------
if [ "$DRY_RUN" = "1" ]; then
echo "[dry-run] Would wait for health at ${HEALTH_URL[$TARGET]} and ${LLM_HEALTH[$TARGET]}"
else
echo "[5/5] Waiting for homepage ${HEALTH_URL[$TARGET]} (via ssh)..."
HEALTH="${HEALTH_URL[$TARGET]}"
for i in $(seq 1 60); do
if ssh "$REMOTE" "curl -sf '$HEALTH'" > /dev/null 2>&1; then
echo " Homepage healthy after approx $((i * 2))s"
break
fi
sleep 2
done
if ! ssh "$REMOTE" "curl -sf '$HEALTH'" > /dev/null 2>&1; then
echo "FAIL: Homepage did not become healthy in 120s"
exit 1
fi
echo "[5/5] Waiting for LLM health ${LLM_HEALTH[$TARGET]} (via ssh)..."
LLM_URL="${LLM_HEALTH[$TARGET]}"
LLM_ITERS="${LLM_WAIT[$TARGET]}"
for i in $(seq 1 "$LLM_ITERS"); do
if ssh "$REMOTE" "curl -sf '$LLM_URL'" > /dev/null 2>&1; then
echo " LLM healthy after approx $((i * 2))s"
echo ""
echo "=== Deploy complete: ${TARGET} ==="
exit 0 exit 0
fi ;;
sleep 2 *)
done echo "Unknown option: $1" >&2
echo "FAIL: LLM service did not become healthy in $((LLM_ITERS * 2))s" usage >&2
exit 2
;;
esac
done
cd "$REPO_ROOT"
if [[ ! -x "$PLANNER" ]]; then
echo "FAIL: missing executable planner: $PLANNER" >&2
exit 1 exit 1
fi fi
# A deployment is a release operation, not a way to publish an arbitrary
# worktree. Include untracked files in this guard; the old diff-only check let
# an untracked secret or half-written config pass the pre-flight.
if [[ -n "$(git status --porcelain)" ]]; then
echo "FAIL: Local working tree has uncommitted changes." >&2
echo "Commit or remove them before deploying." >&2
exit 1
fi
if [[ "$(git branch --show-current)" != "main" ]]; then
echo "FAIL: deploy.sh must run from the main branch." >&2
exit 1
fi
HEAD_REVISION="$(git rev-parse HEAD)"
if BASE_REVISION="$(git rev-parse origin/main 2>/dev/null)"; then
:
else
BASE_REVISION="$(git rev-parse HEAD^ 2>/dev/null || git rev-parse HEAD)"
fi
valid_service() {
local service="$1"
case "$TARGET:$service" in
dankclaw:gluetun|dankclaw:searxng-network|dankclaw:searxng-valkey|dankclaw:searxng|\
dankclaw:pihole|dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
dankclaw:gitea-caddy|dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:caddy|dankclaw:homepage|\
dankclaw:frank-default-large|dankclaw:frank-default-npu|\
dankserver:mariadb|dankserver:redis|dankserver:romm|dankserver:copyparty|\
dankserver:jellyfin|dankserver:calibre-web|dankserver:prowlarr|\
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
dankserver:frank-default-small)
return 0
;;
*)
return 1
;;
esac
}
for service in "${EXPLICIT_SERVICES[@]}"; do
if ! valid_service "$service"; then
echo "Service '$service' is not deployed on $TARGET." >&2
exit 2
fi
done
PLAN=()
if [[ "${#EXPLICIT_SERVICES[@]}" -gt 0 ]]; then
PLAN=("${EXPLICIT_SERVICES[@]}")
else
mapfile -t PLAN < <(
python3 "$PLANNER" --target "$TARGET" --repo "$REPO_ROOT" \
--base "$BASE_REVISION" --head "$HEAD_REVISION"
)
fi
# Preserve planner order while removing duplicate explicit service names.
UNIQUE_PLAN=()
declare -A SEEN=()
for service in "${PLAN[@]}"; do
[[ -n "$service" ]] || continue
if [[ -z "${SEEN[$service]:-}" ]]; then
UNIQUE_PLAN+=("$service")
SEEN[$service]=1
fi
done
PLAN=("${UNIQUE_PLAN[@]}")
printf '=== Deployment plan: %s ===\n' "$TARGET"
printf ' checkout: %s\n' "$HEAD_REVISION"
if [[ "${#PLAN[@]}" -eq 0 ]]; then
echo " changed services: (none; the remote will check pending service state)"
else
printf ' changed services:\n'
printf ' - %s\n' "${PLAN[@]}"
fi
REMOTE="${REMOTE_USER}@${REMOTE_HOST}"
REMOTE_ARGS=("$TARGET" "$HEAD_REVISION")
if [[ "${#EXPLICIT_SERVICES[@]}" -eq 0 ]]; then
REMOTE_ARGS+=(--pending)
fi
REMOTE_ARGS+=("${PLAN[@]}")
if [[ "$DRY_RUN" -eq 1 ]]; then
echo "[dry-run] Would push main to origin."
echo "[dry-run] Would SSH to $REMOTE, pull with --ff-only, and run the service-scoped runner."
printf '[dry-run] Runner arguments:'
printf ' %q' "${REMOTE_ARGS[@]}"
printf '\n'
exit 0
fi
echo "[1/2] Pushing main to origin..."
git push origin main
echo "[2/2] Pulling and deploying on $REMOTE_HOST..."
# The target runner computes any pending revisions from its own state after
# the pull. This keeps a target that missed an earlier service deployment
# recoverable without restarting services that are already current.
ssh "$REMOTE" bash -s -- "${REMOTE_ARGS[@]}" <<'REMOTE_EOF'
set -euo pipefail
TARGET="$1"
EXPECTED_REVISION="$2"
shift 2
cd "/home/dank/serverAppConfigs"
OLD_REVISION="$(git rev-parse HEAD)"
echo " pulling origin/main..."
git pull --ff-only origin main
NEW_REVISION="$(git rev-parse HEAD)"
if [[ "$NEW_REVISION" != "$EXPECTED_REVISION" ]]; then
echo "FAIL: target checkout is $NEW_REVISION, expected $EXPECTED_REVISION" >&2
exit 1
fi
exec ./scripts/deploy-remote.sh "$TARGET" "$NEW_REVISION" "$@"
REMOTE_EOF
echo "=== Deploy complete: $TARGET ==="
scripts/test-changed.sh
+16 -2
View File
@@ -82,9 +82,11 @@ for path in "${CHANGED[@]}"; do
dankserver/compose.yaml|dankclaw/compose.yaml) dankserver/compose.yaml|dankclaw/compose.yaml)
add_test tests/compose_validity.bats add_test tests/compose_validity.bats
add_test tests/env_consistency.bats add_test tests/env_consistency.bats
add_test tests/deployment_scoping.bats
;; ;;
dankserver/Caddyfile|dankclaw/Caddyfile) dankserver/Caddyfile|dankclaw/Caddyfile)
add_test tests/caddyfile.bats add_test tests/caddyfile.bats
add_test tests/deployment_scoping.bats
;; ;;
dankserver/.env.example|dankclaw/.env.example) dankserver/.env.example|dankclaw/.env.example)
add_test tests/env_consistency.bats add_test tests/env_consistency.bats
@@ -95,15 +97,23 @@ for path in "${CHANGED[@]}"; do
add_test tests/homepage_deploy.bats add_test tests/homepage_deploy.bats
add_test tests/homepage_pages.bats add_test tests/homepage_pages.bats
add_test tests/homepage_checker.bats add_test tests/homepage_checker.bats
add_test tests/deployment_scoping.bats
;; ;;
shared/searxng/*) shared/searxng/*)
add_test tests/searxng_limiter.bats add_test tests/searxng_limiter.bats
add_test tests/searxng_quadlet.bats add_test tests/searxng_quadlet.bats
add_test tests/deployment_scoping.bats
;; ;;
dankclaw/*.container|dankclaw/*.network) dankclaw/*.container|dankclaw/*.network)
add_test tests/open_webui_quadlet.bats add_test tests/open_webui_quadlet.bats
add_test tests/pihole.bats add_test tests/pihole.bats
add_test tests/searxng_quadlet.bats add_test tests/searxng_quadlet.bats
add_test tests/deployment_scoping.bats
;;
scripts/deploy.sh|scripts/deploy-remote.sh|scripts/deploy-plan.py|scripts/deploy-*-quadlet.sh)
add_test tests/deployment_flow.bats
add_test tests/deployment_scoping.bats
add_test tests/scripts_syntax.bats
;; ;;
scripts/mount-entropy.sh) scripts/mount-entropy.sh)
add_test tests/mount_entropy.bats add_test tests/mount_entropy.bats
@@ -121,19 +131,23 @@ for path in "${CHANGED[@]}"; do
scripts/*.service) scripts/*.service)
add_test tests/llm_units.bats add_test tests/llm_units.bats
add_test tests/scripts_syntax.bats add_test tests/scripts_syntax.bats
add_test tests/deployment_scoping.bats
;; ;;
scripts/*) scripts/*)
add_test tests/scripts_syntax.bats add_test tests/scripts_syntax.bats
;; ;;
docs/*) docs/*)
add_test tests/docs_completeness.bats add_test tests/docs_completeness.bats
add_test tests/deployment_scoping.bats
;; ;;
dankclaw/hermes/*) dankclaw/hermes/*)
add_test tests/docs_completeness.bats add_test tests/docs_completeness.bats
;; ;;
AGENTS.md) README.md|AGENTS.md)
# The authoring rulebook; validate the docs vault whenever it changes. # Project guidance changes can affect the documented deployment
# contract; validate the docs vault and scoped deployment seams.
add_test tests/docs_completeness.bats add_test tests/docs_completeness.bats
add_test tests/deployment_scoping.bats
;; ;;
*) *)
die "no focused-test mapping for '$path'; add one before committing" die "no focused-test mapping for '$path'; add one before committing"
tests/deployment_flow.bats
+20 -33
View File
@@ -1,8 +1,9 @@
#!/usr/bin/env bats #!/usr/bin/env bats
# Seam: target reachability and deployment flow integrity. # Seam: target reachability and deployment flow integrity.
# These tests assert the deployment contract of the unified repo: # These tests assert the service-scoped deployment contract:
# - a deploy.sh exists and accepts <dankclaw|dankserver> and --dry-run # - deploy.sh accepts a target, optional service scope, and --dry-run
# - deploy.sh covers both the podman stack and the LLM fleet services # - deploy-remote.sh owns per-service handlers and deployment state
# - no host-wide inference restart is hidden in the dispatcher
# - every host referenced in the repo is reachable over SSH (dank@host) # - every host referenced in the repo is reachable over SSH (dank@host)
# - the local git remote is configured before deploy is attempted # - the local git remote is configured before deploy is attempted
# Reachability tests are SKIPPED when the target is unreachable so the suite # Reachability tests are SKIPPED when the target is unreachable so the suite
@@ -38,44 +39,30 @@ load helpers
# Regression: the Caddyfile render step in deploy.sh must survive a bcrypt # Regression: the Caddyfile render step in deploy.sh must survive a bcrypt
# hash (which contains '/' characters). Reproduce the sed expression with a # hash (which contains '/' characters). Reproduce the sed expression with a
# slash-bearing hash and confirm the placeholder is replaced. # slash-bearing hash and confirm the placeholder is replaced.
@test "deploy.sh reloads Caddy after rendering (hot reload, not just container start)" { @test "scoped remote runner owns Caddy reload and safe fallback" {
# The caddy container often survives 'up -d' (a Caddyfile change is a # Caddy is handled only when selected by the planner. A failed reload may
# bind-mount content change, not a container config change). Without an # restart Caddy, but must not restart the whole compose project.
# explicit reload, a rendered fix can sit on disk while Caddy serves the grep -q 'caddy reload' "$REPO_ROOT/scripts/deploy-remote.sh"
# stale in-memory config. grep -q 'podman restart caddy' "$REPO_ROOT/scripts/deploy-remote.sh"
grep -q 'caddy reload' "$REPO_ROOT/scripts/deploy.sh" ! grep -qE 'podman-compose.*(up|restart).*\$\(' "$REPO_ROOT/scripts/deploy-remote.sh"
} }
@test "deploy.sh restarts caddy when the reload cannot be applied" { @test "remote runner has separate handlers for both inference services" {
# If the running config has the admin API disabled (or the API is down), grep -q 'deploy_llm_large' "$REPO_ROOT/scripts/deploy-remote.sh"
# 'caddy reload' can never apply the new file -- the fix cannot be grep -q 'deploy_llm_small' "$REPO_ROOT/scripts/deploy-remote.sh"
# delivered over the API it is meant to enable. The fallback is a
# container restart, which always reads the file.
grep -q 'podman restart caddy' "$REPO_ROOT/scripts/deploy.sh"
} }
# The unified deploy.sh must manage the LLM fleet services on both hosts: @test "inference restarts are absent from the top-level dispatcher" {
# frank-default-small (dankserver) and frank-default-large (dankclaw). A ! grep -qE 'systemctl .*restart (frank-default-small|frank-default-large)' "$REPO_ROOT/scripts/deploy.sh"
# deploy that only restarts the podman stack leaves the LLM units pointing
# at the retired vllmScripts checkout.
@test "deploy.sh restarts the LLM services on both targets" {
grep -q 'frank-default-small' "$REPO_ROOT/scripts/deploy.sh"
grep -q 'frank-default-large' "$REPO_ROOT/scripts/deploy.sh"
} }
@test "deploy.sh health-checks both LLM endpoints (both on :7770, by host)" { @test "remote runner health-checks an inference endpoint" {
# 2026-08-22: frank-default-large moved from :52625 to :7770 on grep -q '127.0.0.1:7770/health' "$REPO_ROOT/scripts/deploy-remote.sh"
# dankclaw. Both LLM services now listen on :7770, distinguished by
# host (dankserver vs dankclaw).
grep -qE '^LLM_HEALTH\[dankserver\]="http://127\.0\.0\.1:7770/health"' "$REPO_ROOT/scripts/deploy.sh"
grep -qE '^LLM_HEALTH\[dankclaw\]="http://127\.0\.0\.1:7770/health"' "$REPO_ROOT/scripts/deploy.sh"
} }
@test "deploy.sh pulls on the merged checkout on both targets" { @test "deploy.sh pulls the merged checkout on the target" {
# Both targets run the whole stack (podman + LLM) from one checkout. grep -q 'git pull --ff-only origin main' "$REPO_ROOT/scripts/deploy.sh"
local count grep -q '/home/dank/serverAppConfigs' "$REPO_ROOT/scripts/deploy.sh"
count=$(grep -c '/home/dank/serverAppConfigs' "$REPO_ROOT/scripts/deploy.sh")
[ "$count" -ge 2 ]
} }
@test "deploy.sh Caddyfile render survives a slash-bearing bcrypt hash" { @test "deploy.sh Caddyfile render survives a slash-bearing bcrypt hash" {
tests/deployment_scoping.bats
+127
View File
@@ -0,0 +1,127 @@
#!/usr/bin/env bats
# Seam: service-scoped deployment planning.
# A repository commit can contain changes for several workloads while only a
# subset is safe to restart. The planner must keep inference isolated from
# unrelated service changes and treat docs-only changes as a no-op.
load helpers
PLANNER="$REPO_ROOT/scripts/deploy-plan.py"
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
@test "deployment planner maps SearXNG changes to SearXNG only" {
run python3 "$PLANNER" --target dankclaw --paths \
shared/searxng/settings.yml docs/Dankclaw/SearXNG.md
[ "$status" -eq 0 ]
[ "$output" = "searxng" ]
}
@test "deployment planner maps large-model changes to the large model only" {
run python3 "$PLANNER" --target dankclaw --paths \
scripts/frank-default-large.env scripts/frank-default-large.service
[ "$status" -eq 0 ]
[ "$output" = "frank-default-large" ]
}
@test "shared chat-template changes map to the selected host's inference service" {
run python3 "$PLANNER" --target dankclaw --paths scripts/chat-template.jinja
[ "$status" -eq 0 ]
[ "$output" = "frank-default-large" ]
run python3 "$PLANNER" --target dankserver --paths scripts/chat-template.jinja
[ "$status" -eq 0 ]
[ "$output" = "frank-default-small" ]
}
@test "deployment planner maps small-model changes to the small model only" {
run python3 "$PLANNER" --target dankserver --paths \
scripts/frank-default-small.service
[ "$status" -eq 0 ]
[ "$output" = "frank-default-small" ]
}
@test "documentation-only changes plan no runtime deployment" {
run python3 "$PLANNER" --target dankclaw --paths \
docs/Architecture/Deployment\ Flow.md docs/Dankclaw/SearXNG.md
[ "$status" -eq 0 ]
[ -z "$output" ]
}
@test "deployment planner maps shared config to one service" {
run python3 "$PLANNER" --target dankclaw --paths \
shared/prometheus/prometheus.yml
[ "$status" -eq 0 ]
[ "$output" = "prometheus" ]
}
@test "deployment planner maps service-owned non-compose files" {
run python3 "$PLANNER" --target dankclaw --paths dankclaw/hermes/config.yaml.example
[ "$status" -eq 0 ]
[ "$output" = "hermes" ]
run python3 "$PLANNER" --target dankclaw --paths scripts/serve-frank-default-npu.sh scripts/frank-default-npu.env
[ "$status" -eq 0 ]
[ "$output" = "frank-default-npu" ]
}
@test "homepage is owned by Dankclaw's consolidated proxy" {
run python3 "$PLANNER" --target dankserver --paths shared/homepage/index.html
[ "$status" -eq 0 ]
[ -z "$output" ]
}
@test "compose planner narrows a compose edit to the changed service" {
local base head
base=$(git -C "$REPO_ROOT" rev-parse HEAD)
head=$(git -C "$REPO_ROOT" rev-parse HEAD)
run python3 "$PLANNER" --target dankclaw --repo "$REPO_ROOT" \
--base "$base" --head "$head" --paths dankclaw/compose.yaml
[ "$status" -eq 0 ]
[ -z "$output" ]
}
@test "network-coupled SearXNG config changes include the required peers" {
run python3 "$PLANNER" --target dankclaw --paths \
dankclaw/searxng/quadlet/gluetun.container
[ "$status" -eq 0 ]
[ "$output" = $'gluetun\nsearxng' ]
}
@test "remote deployer carries per-service deployment state" {
[ -x "$REMOTE_DEPLOY" ]
grep -q 'deploy-state' "$REMOTE_DEPLOY"
grep -q 'STATE\[' "$REMOTE_DEPLOY"
grep -q 'mark_status.*pending\|status=pending' "$REMOTE_DEPLOY"
grep -q 'mark_status.*deployed\|status=deployed' "$REMOTE_DEPLOY"
}
@test "remote deployer routes inference restarts through selected handlers" {
# The runner may restart inference only from the service-dispatch path. It
# must not contain a host-wide restart loop.
grep -q 'deploy_llm_large' "$REMOTE_DEPLOY"
grep -q 'deploy_llm_small' "$REMOTE_DEPLOY"
grep -q 'deploy_service' "$REMOTE_DEPLOY"
! grep -qE 'for .*frank-default-(small|large)' "$REMOTE_DEPLOY"
}
@test "remote runner rejects services belonging to the other host" {
run env DEPLOY_STATE_DIR="$(mktemp -d)" REMOTE_DIR="$REPO_ROOT" \
bash "$REMOTE_DEPLOY" dankserver "$(git rev-parse HEAD)" frank-default-large
[ "$status" -eq 2 ]
}
@test "top-level deployer delegates to the service-scoped remote runner" {
grep -q 'deploy-remote.sh' "$REPO_ROOT/scripts/deploy.sh"
grep -qE 'SERVICE|service' "$REPO_ROOT/scripts/deploy.sh"
grep -q 'OLD_REVISION' "$REPO_ROOT/scripts/deploy.sh"
grep -q 'OLD_REVISION\|old checkout revision' "$REPO_ROOT/scripts/deploy.sh"
}
@test "legacy service deploy wrappers delegate to the scoped deployer" {
for script in deploy-searxng-quadlet.sh deploy-pihole-quadlet.sh deploy-open-webui-quadlet.sh; do
grep -q 'deploy.sh' "$REPO_ROOT/scripts/$script" || {
echo "$script does not delegate to deploy.sh"
return 1
}
done
}
tests/homepage_deploy.bats
+7 -5
View File
@@ -10,10 +10,12 @@ load helpers
@test "deploy.sh ships the homepage (assets, timer, basicauth hash, deployed.js)" { @test "deploy.sh ships the homepage (assets, timer, basicauth hash, deployed.js)" {
local d="$REPO_ROOT/scripts/deploy.sh" local d="$REPO_ROOT/scripts/deploy.sh"
grep -q "/srv/homepage" "$d" local r="$REPO_ROOT/scripts/deploy-remote.sh"
grep -q "homepage-status" "$d" grep -q "deploy-remote.sh" "$d"
grep -q "CADDY_BASICAUTH_HASH" "$d" grep -q "/srv/homepage" "$r"
grep -q "deployed.js" "$d" grep -q "homepage-status" "$r"
grep -q "CADDY_BASICAUTH_HASH" "$r"
grep -q "deployed.js" "$r"
} }
@test "the checker runs as a systemd timer pointing at the checker" { @test "the checker runs as a systemd timer pointing at the checker" {
@@ -27,5 +29,5 @@ load helpers
} }
@test "deploy.sh is git-driven: it pulls main on the target" { @test "deploy.sh is git-driven: it pulls main on the target" {
grep -q "git pull origin main" "$REPO_ROOT/scripts/deploy.sh" grep -q "git pull --ff-only origin main" "$REPO_ROOT/scripts/deploy.sh"
} }
tests/searxng_limiter.bats
+5 -4
View File
@@ -90,8 +90,9 @@ PYEOF
} }
@test "deploy.sh copies limiter.toml into /opt/searxng/config" { @test "deploy.sh copies limiter.toml into /opt/searxng/config" {
grep -q "shared/searxng/limiter.toml" "$REPO_ROOT/scripts/deploy.sh" \ local d="$REPO_ROOT/scripts/deploy-remote.sh"
|| { echo "deploy.sh does not copy shared/searxng/limiter.toml"; return 1; } grep -q "shared/searxng/limiter.toml" "$d" \
grep -q "/opt/searxng/config/limiter.toml" "$REPO_ROOT/scripts/deploy.sh" \ || { echo "deploy-remote.sh does not copy shared/searxng/limiter.toml"; return 1; }
|| { echo "deploy.sh does not place limiter.toml at /opt/searxng/config"; return 1; } grep -q "/opt/searxng/config/limiter.toml" "$d" \
|| { echo "deploy-remote.sh does not place limiter.toml at /opt/searxng/config"; return 1; }
} }