llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
This commit is contained in:
@@ -94,7 +94,7 @@ earlier can still hold listener/TIME_WAIT state, so the replacement dies
|
||||
`pasta: ... bind: Address already in use` (conmon exit 126) — then **self-heals
|
||||
seconds later** under `Restart=on-failure`. The failure looks like it failed,
|
||||
then looks fine, and the missing gate stays invisible because the retry masks
|
||||
it. Observed 2026-09-16 cutting `frank-large` from a root unit to a rootless
|
||||
it. Observed 2026-09-16 cutting `frank-dragoon` from a root unit to a rootless
|
||||
Quadlet.
|
||||
|
||||
Gate procedure: stop the old unit AND the new one (a redeploy's running
|
||||
|
||||
@@ -217,8 +217,8 @@ done
|
||||
```
|
||||
|
||||
Ground truth (verify, don't assume — IPs can move): 192.168.0.143 = dankclaw
|
||||
= AI stack + `frank-large` :7770 + `frank-secondary` :7771 (RTX 3090 + Strix
|
||||
Halo iGPU); 192.168.0.235 = dankserver = media stack + `frank-primary`
|
||||
= AI stack + `frank-dragoon` :7770 + `frank-zealot` :7771 (RTX 3090 + Strix
|
||||
Halo iGPU); 192.168.0.235 = dankserver = media stack + `frank-archon`
|
||||
:7770 (2× R9700). Repo dirs + live hostnames agree; the deploy tooling is
|
||||
hostname-anchored (`ssh dankclaw`/`ssh dankserver` via `~/.ssh/config`), so
|
||||
it is unaffected by IP drift.
|
||||
|
||||
@@ -12,7 +12,7 @@ timestamp: 2026-10-05T00:00:00-07:00
|
||||
|
||||
Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
|
||||
service was not crashed. Applies to `frank-medium` (dankserver, root unit),
|
||||
`frank-secondary` (dankclaw, root unit), `frank-large` (dankclaw, rootless
|
||||
`frank-zealot` (dankclaw, root unit), `frank-dragoon` (dankclaw, rootless
|
||||
Quadlet).
|
||||
|
||||
**1. Classify the stop before touching anything**
|
||||
@@ -95,22 +95,22 @@ the local Gitea on Dankclaw — push allowed).
|
||||
unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
|
||||
freshly restarted endpoint under live agent-queue backlog may return empty
|
||||
bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
|
||||
timeout. First request after a halogen (`frank-large`) restart can go silent
|
||||
45–60 s and self-recover; do not restart over it. Rootless `frank-large`:
|
||||
timeout. First request after a halogen (`frank-dragoon`) restart can go silent
|
||||
45–60 s and self-recover; do not restart over it. Rootless `frank-dragoon`:
|
||||
read logs **without** sudo (per-user storage). If the restore never happened,
|
||||
the sudo journal names who and when: `journalctl -b | grep sudo`.
|
||||
|
||||
## Scratch vLLM boot on a fleet GPU host — stop-first law
|
||||
|
||||
Any scratch/experimental vLLM container on a host that also runs a production
|
||||
LLM service (`frank-primary`, `frank-large`, `frank-secondary`) MUST stop the
|
||||
LLM service (`frank-archon`, `frank-dragoon`, `frank-zealot`) MUST stop the
|
||||
production unit first, and the window script must verify it. Two independent
|
||||
failure modes if skipped:
|
||||
|
||||
1. **Expert-offload planner dies on host RAM.**
|
||||
`vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at
|
||||
plan time. A production rank worker holds tens of GiB of unevictable
|
||||
pinned expert rows (frank-primary: ~76 GiB × 2 ranks), so a beside-prod
|
||||
pinned expert rows (frank-archon: ~76 GiB × 2 ranks), so a beside-prod
|
||||
boot sees ~86 GiB instead of ~239 and dies at plan time:
|
||||
`ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`.
|
||||
Looks like a planner bug; is pure RAM contention. Signature: container
|
||||
@@ -124,15 +124,15 @@ failure modes if skipped:
|
||||
Window skeleton (dankserver):
|
||||
|
||||
```bash
|
||||
sudo systemctl stop frank-primary.service
|
||||
sudo systemctl mask --runtime frank-primary.service # relinked unit: plain mask refuses; runtime mask self-clears at reboot
|
||||
sudo systemctl stop frank-archon.service
|
||||
sudo systemctl mask --runtime frank-archon.service # relinked unit: plain mask refuses; runtime mask self-clears at reboot
|
||||
sleep 3
|
||||
podman ps --format "{{.Names}}" | grep -q frank-primary-glm && { echo "prod still running - abort"; exit 1; }
|
||||
podman ps --format "{{.Names}}" | grep -q frank-archon-glm && { echo "prod still running - abort"; exit 1; }
|
||||
# ... launch scratch container, watch logs ...
|
||||
podman rm -f <scratch> # stuck vLLM ignores SIGTERM
|
||||
sudo systemctl unmask --runtime frank-primary.service
|
||||
sudo systemctl reset-failed frank-primary.service
|
||||
sudo systemctl start frank-primary.service
|
||||
sudo systemctl unmask --runtime frank-archon.service
|
||||
sudo systemctl reset-failed frank-archon.service
|
||||
sudo systemctl start frank-archon.service
|
||||
# restore oracle is a CONJUNCTION: health 200 AND is-active active AND is-enabled enabled AND no /run/systemd/system mask symlink
|
||||
```
|
||||
|
||||
|
||||
@@ -49,7 +49,7 @@ table. Three buckets:
|
||||
- **retired** — maps to a retired runtime. Safe candidates.
|
||||
|
||||
Trap: a directory can hold **both** a dead target and a live drafter
|
||||
(frank-large's DFlash2 drafter lives beside the previous target). Move the
|
||||
(frank-dragoon's DFlash2 drafter lives beside the previous target). Move the
|
||||
single file, never the directory, and say so in the manifest.
|
||||
|
||||
## 3. The live-reference check — before moving anything
|
||||
@@ -64,7 +64,7 @@ done
|
||||
Then **read every hit**; do not count them. Most are benign (comments,
|
||||
retired fetch-script destinations, `retire_*_quadlet` helpers that move a
|
||||
quadlet not a model) — but on Dankclaw this check found a live bug:
|
||||
`serve-frank-secondary.sh`'s `: "${MODEL_DIR:=...}"` fallback still pointed at a
|
||||
`serve-frank-zealot.sh`'s `: "${MODEL_DIR:=...}"` fallback still pointed at a
|
||||
payload retired a week earlier. The env file always overrode it, so it was
|
||||
invisible; a missing env file would have produced a "missing asset" naming a
|
||||
model that no longer exists.
|
||||
@@ -99,7 +99,7 @@ active).
|
||||
## 5. Verify, then report — and do not purge
|
||||
|
||||
```sh
|
||||
systemctl is-active frank-large frank-secondary
|
||||
systemctl is-active frank-dragoon frank-zealot
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health
|
||||
```
|
||||
|
||||
@@ -149,7 +149,7 @@ name**, never `endswith('.gguf')` — mradermacher ships large quants as
|
||||
|
||||
## Refresh per engine — know the recovery path BEFORE deleting
|
||||
|
||||
- **frank-large (halogen)**: the engine fetches its own payload via
|
||||
- **frank-dragoon (halogen)**: the engine fetches its own payload via
|
||||
`HALOGEN_DOWNLOAD` when the checkpoint is absent. Clear the directory but
|
||||
leave it **existing and empty**, then start the unit. **Bind-mount trap:**
|
||||
podman refuses a mount whose source is missing
|
||||
|
||||
Reference in New Issue
Block a user