feat: add consistent deployment/update process (deploy.sh + docs + AGENTS.md)

- scripts/deploy.sh: target-aware deploy (dankclaw/dankserver). Pushes main, pulls on the target, installs/enables systemd unit, restarts service, and checks the health endpoint; --dry-run preview; pre-flight dirty-tree check - docs/deployment.md: full deployment/update process with mermaid workflow, target map, manual update commands, service units, and troubleshooting - AGENTS.md: add Deployment & Update Process section (deploy.sh usage, target map, mermaid workflow) and a Summary table row - docs/index.md: crosslink deployment doc
2026-08-10 21:04:37 -07:00
parent 3a64c8f3dd
commit 90c4883f89
4 changed files with 314 additions and 3 deletions
@@ -1,8 +1,10 @@
 # AGENTS.md — vllmScripts Agent Rules

-## Git Workflow
+## Overview

-This project manages deployment scripts for vLLM and llama.cpp servers on dankserver and dankclaw.
+This project manages deployment scripts for two LLM inference servers. Dankclaw runs a ROCmFPX llama.cpp service. Dankserver runs a vLLM service. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process.
+
+## Git Workflow

 All script edits must flow through git. Do not edit files on the remote machine directly.

@@ -15,7 +17,35 @@ All script edits must flow through git. Do not edit files on the remote machine
 5. **Pull on the target** — `git pull` on the remote machine before restarting
 6. **Restart the service** — use the updated script on the target

-### Why
+## Deployment & Update Process
+
+Use `scripts/deploy.sh` to deploy committed changes to a target. It pushes `main`, pulls on the target, restarts the service, and checks the health endpoint.
+
+```
+./scripts/deploy.sh dankclaw        # ROCmFPX llama.cpp (llama-server, port 8080)
+./scripts/deploy.sh dankserver      # vLLM (vllm-qwen36, port 8000)
+./scripts/deploy.sh dankclaw --dry-run   # preview without changes
+```
+
+Deployment workflow:
+
+```mermaid
+graph LR
+    A[Edit locally] --> B[Commit]
+    B --> C[Push to origin]
+    C --> D[deploy.sh: pull on target]
+    D --> E[restart service]
+    E --> F[health check]
+```
+
+Target map:
+
+| Target | Host | Runtime | Service | Port |
+|--------|------|---------|---------|------|
+| Dankclaw | dank@dankclaw | ROCmFPX llama.cpp | llama-server | 8080 |
+| Dankserver | dank@dankserver | vLLM (Docker) | vllm-qwen36 | 8000 |
+
+## Why

 Direct edits on the remote drift from version control. Future agents will not know what changed. Git is the single source of truth.

@@ -28,4 +58,5 @@ Direct edits on the remote drift from version control. Future agents will not kn
 | Editing a deployment script | **Work through git** — fetch, commit, edit, push, pull, restart. Never edit on the remote directly. |
 | Script exists only on remote | **Pull it into the repo first** — commit the fetched version before making changes. |
 | Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. |
+| Deploying a committed change | **Run `scripts/deploy.sh <target>`** — it pushes, pulls on the target, restarts the service, and checks health. See [docs/deployment.md](docs/deployment.md). |
 | Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
@@ -0,0 +1,133 @@
+# Deployment and Update Process
+
+This page describes how to deploy and update the LLM servers on Dankserver and Dankclaw. The process uses git as the single source of truth. All script edits flow through this repository.
+
+## Overview
+
+Two servers run LLM inference services. Each server pulls its scripts from this repository. The deployment process moves a committed change from the repository to a running service.
+
+| Target | Host | Runtime | Service | Port |
+|--------|------|---------|---------|------|
+| Dankclaw | dank@dankclaw | ROCmFPX llama.cpp | llama-server | 8080 |
+| Dankserver | dank@dankserver | vLLM (Docker) | vllm-qwen36 | 8000 |
+
+## Workflow
+
+The deployment workflow has six steps.
+
+```mermaid
+graph LR
+    A[Edit scripts locally] --> B[Commit]
+    B --> C[Push to origin<br>Gitea on Dankclaw]
+    C --> D[Pull on target<br>git pull]
+    D --> E[Restart systemd service]
+    E --> F[Verify health endpoint]
+```
+
+1. Fetch the current script from the target machine.
+2. Commit the fetched version in this repository.
+3. Edit the script locally in the working tree.
+4. Commit and push the change to origin.
+5. Pull the change on the target machine.
+6. Restart the service and verify that it is healthy.
+
+## Deploy Script
+
+The `scripts/deploy.sh` script executes steps 4 through 6. Run it from the repository root.
+
+Run this command to deploy to Dankclaw:
+
+```
+./scripts/deploy.sh dankclaw
+```
+
+Run this command to deploy to Dankserver:
+
+```
+./scripts/deploy.sh dankserver
+```
+
+Add the `--dry-run` flag to preview without making changes:
+
+```
+./scripts/deploy.sh dankclaw --dry-run
+```
+
+The script does these actions:
+
+1. Verifies that the local working tree has no uncommitted changes.
+2. Pushes the `main` branch to origin.
+3. Pulls the repository on the target machine.
+4. Installs or updates the systemd unit.
+5. Reloads systemd and restarts the service.
+6. Waits for the health endpoint to respond.
+
+## Manual Update
+
+If you cannot run the deploy script, update a server manually. Complete each step on the target machine.
+
+### Update Dankclaw
+
+1. Pull the repository.
+2. Symlink the unit file, if it changed.
+3. Reload and restart the service.
+
+Run these commands on Dankclaw:
+
+```
+cd /home/dank/vllmScripts
+git pull origin main
+sudo ln -sf /home/dank/vllmScripts/scripts/llama-server.service /etc/systemd/system/llama-server.service
+sudo systemctl daemon-reload
+sudo systemctl restart llama-server
+curl -sf http://127.0.0.1:8080/health
+```
+
+### Update Dankserver
+
+Run these commands on Dankserver:
+
+```
+cd /home/dank/vLLM
+git pull origin main
+sudo systemctl daemon-reload
+sudo systemctl restart vllm-qwen36
+curl -sf http://127.0.0.1:8000/health
+```
+
+## Service Units
+
+### Dankclaw — llama-server
+
+The active service on Dankclaw runs the ROCmFPX llama.cpp runtime. It serves the Ling-3.0-flash ROCmFP4 MTP model with `draft-mtp` speculative decoding. The unit file is `scripts/llama-server.service` in the repository. The machine symlinks it to `/etc/systemd/system/llama-server.service`.
+
+### Dankserver — vllm-qwen36
+
+The vLLM service on Dankserver runs in a Docker container. The unit file is a physical file on the machine, not a symlink into the repository. The `vllm-qwen36.sh` launcher in the repository passes the model and flags to `docker run`.
+
+> Note: a second unit, `vllm-awq-qwen36.service`, currently runs the AWQ 4-bit model on port 34401. Verify the intended service before a restart. See the repository `git log` for the latest decision.
+
+## Validation
+
+The deploy script checks the health endpoint after a restart. For deeper checks, run the validation script on the target.
+
+Run this command on Dankserver to test vLLM throughput:
+
+```
+./scripts/validate-vllm.sh
+```
+
+## Troubleshooting
+
+If the health check fails after a deploy:
+
+1. Read the service logs with `journalctl -u <service> -n 200`.
+2. Confirm that the model files exist in the cache.
+3. Confirm that the GPU has free memory.
+4. Check the service status with `systemctl status <service>`.
+
+## See Also
+
+- [AGENTS.md](../AGENTS.md) — the git workflow rules
+- [README.md](../README.md) — repository overview
+- [index.md](index.md) — knowledge base index
@@ -35,6 +35,7 @@ graph LR

 - [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
 - [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
+- [[deployment]] — deploy and update process for both servers

 ## Memory Management

@@ -0,0 +1,146 @@
+#!/usr/bin/env bash
+# deploy.sh — Deploy the vllmScripts repo to a target and restart its service.
+#
+# This script implements the second half of the git deployment workflow from
+# AGENTS.md. It assumes the local changes are committed and pushed, then:
+#   1. Pulls the repo on the target
+#   2. Installs / enables the systemd unit
+#   3. Reloads systemd and restarts the service
+#   4. Waits for the health endpoint and reports status
+#
+# Usage:
+#   ./deploy.sh <target> [--dry-run]
+#
+# Targets:
+#   dankclaw   - ROCmFPX llama.cpp server (llama-server.service, port 8080)
+#   dankserver - vLLM server (vllm-qwen36.service, port 8000)
+#
+# Example:
+#   ./deploy.sh dankclaw
+#   ./deploy.sh dankserver --dry-run
+set -euo pipefail
+
+# ---------------------------------------------------------------------------
+# Target configuration
+# ---------------------------------------------------------------------------
+declare -A REMOTE_USER REMOTE_HOST REMOTE_DIR SERVICE SERVICE_HEALTH
+
+REMOTE_USER[dankclaw]=dank
+REMOTE_HOST[dankclaw]=dankclaw
+REMOTE_DIR[dankclaw]="/home/dank/vllmScripts"
+SERVICE[dankclaw]="llama-server"
+SERVICE_HEALTH[dankclaw]="http://127.0.0.1:8080/health"
+
+REMOTE_USER[dankserver]=dank
+REMOTE_HOST[dankserver]=dankserver
+REMOTE_DIR[dankserver]="/home/dank/vLLM"
+SERVICE[dankserver]="vllm-qwen36"
+SERVICE_HEALTH[dankserver]="http://127.0.0.1:8000/health"
+
+TARGET="${1:-}"
+DRY_RUN=0
+if [ "${2:-}" = "--dry-run" ]; then
+    DRY_RUN=1
+fi
+
+# ---------------------------------------------------------------------------
+# Validation
+# ---------------------------------------------------------------------------
+if [ -z "${TARGET:-}" ]; then
+    echo "Usage: $0 <dankclaw|dankserver> [--dry-run]"
+    exit 1
+fi
+if [ -z "${REMOTE_HOST[$TARGET]:-}" ]; then
+    echo "Unknown target: $TARGET"
+    exit 1
+fi
+
+# ---------------------------------------------------------------------------
+# Local checks (pre-flight)
+# ---------------------------------------------------------------------------
+cd "$(dirname "$0")/.."
+if ! git diff --quiet HEAD; then
+    echo "FAIL: Local working tree has uncommitted changes."
+    echo "Commit or stash them before deploying."
+    exit 1
+fi
+if [ "$(git rev-parse --abbrev-ref HEAD)" != "main" ]; then
+    echo "WARN: Not on 'main'; deploying a non-main branch."
+fi
+
+echo "=== Deploying to ${TARGET} ==="
+echo "  host:    ${REMOTE_HOST[$TARGET]}"
+echo "  dir:     ${REMOTE_DIR[$TARGET]}"
+echo "  service: ${SERVICE[$TARGET]}"
+echo "  health:  ${SERVICE_HEALTH[$TARGET]}"
+echo ""
+
+# ---------------------------------------------------------------------------
+# Push (unless dry-run)
+# ---------------------------------------------------------------------------
+if [ "$DRY_RUN" = "1" ]; then
+    echo "[dry-run] Would push local main to origin"
+    echo "[dry-run] Would SSH to ${REMOTE_HOST[$TARGET]} and git pull"
+else
+    echo "[1/4] Pushing local main to origin..."
+    git push origin main
+fi
+
+REMOTE="$(
+    printf '%s@%s' "${REMOTE_USER[$TARGET]}" "${REMOTE_HOST[$TARGET]}"
+)"
+
+# ---------------------------------------------------------------------------
+# Remote: pull + install + restart
+# ---------------------------------------------------------------------------
+REMOTE_SCRIPT="
+set -euo pipefail
+cd '${REMOTE_DIR[$TARGET]}'
+
+echo '[2/4] Pulling repo on ${TARGET}...'
+git pull origin main
+
+# Install / symlink the systemd unit
+case '${TARGET}' in
+  dankclaw)
+    sudo ln -sf '${REMOTE_DIR[$TARGET]}/scripts/llama-server.service' /etc/systemd/system/llama-server.service
+    ;;
+  dankserver)
+    # vllm-qwen36.service is a physical file; no symlink needed.
+    :
+    ;;
+esac
+
+echo '[3/4] Reload and restart ${SERVICE[$TARGET]}...'
+sudo systemctl daemon-reload
+sudo systemctl enable '${SERVICE[$TARGET]}'
+sudo systemctl restart '${SERVICE[$TARGET]}'
+"
+
+if [ "$DRY_RUN" = "1" ]; then
+    echo "[dry-run] Remote commands would run:"
+    echo "$REMOTE_SCRIPT"
+else
+    ssh "$REMOTE" "bash -c '$REMOTE_SCRIPT'"
+fi
+
+# ---------------------------------------------------------------------------
+# Health check
+# ---------------------------------------------------------------------------
+if [ "$DRY_RUN" = "1" ]; then
+    echo "[dry-run] Would wait for health at ${SERVICE_HEALTH[$TARGET]}"
+else
+    echo "[4/4] Waiting for health endpoint ${SERVICE_HEALTH[$TARGET]}..."
+    HEALTH_URL="${SERVICE_HEALTH[$TARGET]}"
+    for i in $(seq 1 300); do
+        if curl -sf "$HEALTH_URL" > /dev/null 2>&1; then
+            echo "  Service healthy after approx $((i * 2))s"
+            echo ""
+            echo "=== Deploy complete: ${TARGET} ==="
+            exit 0
+        fi
+        sleep 2
+    done
+    echo "FAIL: Service did not become healthy in 600s"
+    exit 1
+fi