llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot

The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.

Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
This commit is contained in:
Frank
2026-10-10 17:03:14 -07:00
parent 5ced474eb9
commit 55f87fcdce
195 changed files with 2085 additions and 2064 deletions
@@ -94,7 +94,7 @@ earlier can still hold listener/TIME_WAIT state, so the replacement dies
`pasta: ... bind: Address already in use` (conmon exit 126) — then **self-heals
seconds later** under `Restart=on-failure`. The failure looks like it failed,
then looks fine, and the missing gate stays invisible because the retry masks
it. Observed 2026-09-16 cutting `frank-large` from a root unit to a rootless
it. Observed 2026-09-16 cutting `frank-dragoon` from a root unit to a rootless
Quadlet.
Gate procedure: stop the old unit AND the new one (a redeploy's running
@@ -217,8 +217,8 @@ done
```
Ground truth (verify, don't assume — IPs can move): 192.168.0.143 = dankclaw
= AI stack + `frank-large` :7770 + `frank-secondary` :7771 (RTX 3090 + Strix
Halo iGPU); 192.168.0.235 = dankserver = media stack + `frank-primary`
= AI stack + `frank-dragoon` :7770 + `frank-zealot` :7771 (RTX 3090 + Strix
Halo iGPU); 192.168.0.235 = dankserver = media stack + `frank-archon`
:7770 (2× R9700). Repo dirs + live hostnames agree; the deploy tooling is
hostname-anchored (`ssh dankclaw`/`ssh dankserver` via `~/.ssh/config`), so
it is unaffected by IP drift.
@@ -12,7 +12,7 @@ timestamp: 2026-10-05T00:00:00-07:00
Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
service was not crashed. Applies to `frank-medium` (dankserver, root unit),
`frank-secondary` (dankclaw, root unit), `frank-large` (dankclaw, rootless
`frank-zealot` (dankclaw, root unit), `frank-dragoon` (dankclaw, rootless
Quadlet).
**1. Classify the stop before touching anything**
@@ -95,22 +95,22 @@ the local Gitea on Dankclaw — push allowed).
unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
freshly restarted endpoint under live agent-queue backlog may return empty
bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
timeout. First request after a halogen (`frank-large`) restart can go silent
45–60 s and self-recover; do not restart over it. Rootless `frank-large`:
timeout. First request after a halogen (`frank-dragoon`) restart can go silent
45–60 s and self-recover; do not restart over it. Rootless `frank-dragoon`:
read logs **without** sudo (per-user storage). If the restore never happened,
the sudo journal names who and when: `journalctl -b | grep sudo`.
## Scratch vLLM boot on a fleet GPU host — stop-first law
Any scratch/experimental vLLM container on a host that also runs a production
LLM service (`frank-primary`, `frank-large`, `frank-secondary`) MUST stop the
LLM service (`frank-archon`, `frank-dragoon`, `frank-zealot`) MUST stop the
production unit first, and the window script must verify it. Two independent
failure modes if skipped:
1. **Expert-offload planner dies on host RAM.**
`vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at
plan time. A production rank worker holds tens of GiB of unevictable
pinned expert rows (frank-primary: ~76 GiB × 2 ranks), so a beside-prod
pinned expert rows (frank-archon: ~76 GiB × 2 ranks), so a beside-prod
boot sees ~86 GiB instead of ~239 and dies at plan time:
`ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`.
Looks like a planner bug; is pure RAM contention. Signature: container
@@ -124,15 +124,15 @@ failure modes if skipped:
Window skeleton (dankserver):
```bash
sudo systemctl stop frank-primary.service
sudo systemctl mask --runtime frank-primary.service # relinked unit: plain mask refuses; runtime mask self-clears at reboot
sudo systemctl stop frank-archon.service
sudo systemctl mask --runtime frank-archon.service # relinked unit: plain mask refuses; runtime mask self-clears at reboot
sleep 3
podman ps --format "{{.Names}}" | grep -q frank-primary-glm && { echo "prod still running - abort"; exit 1; }
podman ps --format "{{.Names}}" | grep -q frank-archon-glm && { echo "prod still running - abort"; exit 1; }
# ... launch scratch container, watch logs ...
podman rm -f <scratch> # stuck vLLM ignores SIGTERM
sudo systemctl unmask --runtime frank-primary.service
sudo systemctl reset-failed frank-primary.service
sudo systemctl start frank-primary.service
sudo systemctl unmask --runtime frank-archon.service
sudo systemctl reset-failed frank-archon.service
sudo systemctl start frank-archon.service
# restore oracle is a CONJUNCTION: health 200 AND is-active active AND is-enabled enabled AND no /run/systemd/system mask symlink
```
@@ -49,7 +49,7 @@ table. Three buckets:
- **retired** — maps to a retired runtime. Safe candidates.
Trap: a directory can hold **both** a dead target and a live drafter
(frank-large's DFlash2 drafter lives beside the previous target). Move the
(frank-dragoon's DFlash2 drafter lives beside the previous target). Move the
single file, never the directory, and say so in the manifest.
## 3. The live-reference check — before moving anything
@@ -64,7 +64,7 @@ done
Then **read every hit**; do not count them. Most are benign (comments,
retired fetch-script destinations, `retire_*_quadlet` helpers that move a
quadlet not a model) — but on Dankclaw this check found a live bug:
`serve-frank-secondary.sh`'s `: "${MODEL_DIR:=...}"` fallback still pointed at a
`serve-frank-zealot.sh`'s `: "${MODEL_DIR:=...}"` fallback still pointed at a
payload retired a week earlier. The env file always overrode it, so it was
invisible; a missing env file would have produced a "missing asset" naming a
model that no longer exists.
@@ -99,7 +99,7 @@ active).
## 5. Verify, then report — and do not purge
```sh
systemctl is-active frank-large frank-secondary
systemctl is-active frank-dragoon frank-zealot
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health
```
@@ -149,7 +149,7 @@ name**, never `endswith('.gguf')` — mradermacher ships large quants as
## Refresh per engine — know the recovery path BEFORE deleting
- **frank-large (halogen)**: the engine fetches its own payload via
- **frank-dragoon (halogen)**: the engine fetches its own payload via
`HALOGEN_DOWNLOAD` when the checkpoint is absent. Clear the directory but
leave it **existing and empty**, then start the unit. **Bind-mount trap:**
podman refuses a mount whose source is missing