llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
This commit is contained in:
@@ -12,7 +12,7 @@ timestamp: 2026-10-05T00:00:00-07:00
|
||||
|
||||
Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
|
||||
service was not crashed. Applies to `frank-medium` (dankserver, root unit),
|
||||
`frank-secondary` (dankclaw, root unit), `frank-large` (dankclaw, rootless
|
||||
`frank-zealot` (dankclaw, root unit), `frank-dragoon` (dankclaw, rootless
|
||||
Quadlet).
|
||||
|
||||
**1. Classify the stop before touching anything**
|
||||
@@ -95,22 +95,22 @@ the local Gitea on Dankclaw — push allowed).
|
||||
unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
|
||||
freshly restarted endpoint under live agent-queue backlog may return empty
|
||||
bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
|
||||
timeout. First request after a halogen (`frank-large`) restart can go silent
|
||||
45–60 s and self-recover; do not restart over it. Rootless `frank-large`:
|
||||
timeout. First request after a halogen (`frank-dragoon`) restart can go silent
|
||||
45–60 s and self-recover; do not restart over it. Rootless `frank-dragoon`:
|
||||
read logs **without** sudo (per-user storage). If the restore never happened,
|
||||
the sudo journal names who and when: `journalctl -b | grep sudo`.
|
||||
|
||||
## Scratch vLLM boot on a fleet GPU host — stop-first law
|
||||
|
||||
Any scratch/experimental vLLM container on a host that also runs a production
|
||||
LLM service (`frank-primary`, `frank-large`, `frank-secondary`) MUST stop the
|
||||
LLM service (`frank-archon`, `frank-dragoon`, `frank-zealot`) MUST stop the
|
||||
production unit first, and the window script must verify it. Two independent
|
||||
failure modes if skipped:
|
||||
|
||||
1. **Expert-offload planner dies on host RAM.**
|
||||
`vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at
|
||||
plan time. A production rank worker holds tens of GiB of unevictable
|
||||
pinned expert rows (frank-primary: ~76 GiB × 2 ranks), so a beside-prod
|
||||
pinned expert rows (frank-archon: ~76 GiB × 2 ranks), so a beside-prod
|
||||
boot sees ~86 GiB instead of ~239 and dies at plan time:
|
||||
`ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`.
|
||||
Looks like a planner bug; is pure RAM contention. Signature: container
|
||||
@@ -124,15 +124,15 @@ failure modes if skipped:
|
||||
Window skeleton (dankserver):
|
||||
|
||||
```bash
|
||||
sudo systemctl stop frank-primary.service
|
||||
sudo systemctl mask --runtime frank-primary.service # relinked unit: plain mask refuses; runtime mask self-clears at reboot
|
||||
sudo systemctl stop frank-archon.service
|
||||
sudo systemctl mask --runtime frank-archon.service # relinked unit: plain mask refuses; runtime mask self-clears at reboot
|
||||
sleep 3
|
||||
podman ps --format "{{.Names}}" | grep -q frank-primary-glm && { echo "prod still running - abort"; exit 1; }
|
||||
podman ps --format "{{.Names}}" | grep -q frank-archon-glm && { echo "prod still running - abort"; exit 1; }
|
||||
# ... launch scratch container, watch logs ...
|
||||
podman rm -f <scratch> # stuck vLLM ignores SIGTERM
|
||||
sudo systemctl unmask --runtime frank-primary.service
|
||||
sudo systemctl reset-failed frank-primary.service
|
||||
sudo systemctl start frank-primary.service
|
||||
sudo systemctl unmask --runtime frank-archon.service
|
||||
sudo systemctl reset-failed frank-archon.service
|
||||
sudo systemctl start frank-archon.service
|
||||
# restore oracle is a CONJUNCTION: health 200 AND is-active active AND is-enabled enabled AND no /run/systemd/system mask symlink
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user