Frank
a34322aeb0
The managed-skill library was regenerated: 94 narrow per-incident skills
replaced by 25 broader ones (radiance-*, llamacpp-*, homepage-*, fleet/
endpoint ops). Git renders this as mass delete + mass add because whole
directories changed name; old content stays in history at 12c5e88.
Surface check: the 25 on disk are exactly the skills the harness advertises
this session.
28 lines
2.2 KiB
Markdown
28 lines
2.2 KiB
Markdown
---
|
|
name: llamacpp-serve-unit-boot-loop
|
|
description: "Diagnose a Dank-fleet LLM systemd unit that boot-loops: classify 203/EXEC vs model-load OOM vs payload gate, using journal signatures and nvidia-smi/rocm-smi."
|
|
---
|
|
|
|
# llama.cpp / fleet serve unit boot loop
|
|
|
|
A fleet LLM unit (`frank-large`, `frank-secondary`, `frank-primary`) restarts every `RestartSec`. Read the journal FIRST, act second:
|
|
|
|
```
|
|
ssh <host> 'sudo systemctl status <svc> --no-pager | head -12; sudo journalctl -u <svc> --since "-6 min" -o cat --no-pager | tail -40'
|
|
```
|
|
|
|
## Classify by signature
|
|
|
|
| Signature | Cause | Fix |
|
|
|---|---|---|
|
|
| `status=203/EXEC` + `Failed at step EXEC spawning` + `Mem peak: ~1.9M` + GPU flat at 1 MiB | the `ExecStart` script never ran: missing shebang, lost exec bit, or path moved by the deploy (git pull mid-restart) | `head -1 <script>` must be `#!/usr/bin/env bash`; `git ls-files -s <script>` must show `100755`; commit the fix, redeploy |
|
|
| `status=203/EXEC` right after a deploy, then self-heals | git replaced the file mid-checkout while the restart loop ran | ignore if the newest attempts succeed; verify `podman ps --format '{{.CreatedAt}}'` identity |
|
|
| Engine prints weights loading, then dies | VRAM/disk budget, engine flags | read the engine's own ledger (verbosity 4) for KV/compute-buffer MiB; price from two boots, never a formula |
|
|
| Health 503, no error, minutes of silence | normal cold load (big PLE/mmap) | do not kill; the listener binds after the full load |
|
|
|
|
## Rules
|
|
- Judge the newest container/attempt only: `podman ps -a --format '{{.ID}} {{.CreatedAt}}'`, `journalctl --since`; a mid-stop container still serves old logs.
|
|
- `systemctl is-active` is `failed` after every intentional stop on foreground-podman units (exit 143) — cosmetic.
|
|
- The unit's `Description` is the last-deployed text, not the repo's. A stale Description is not an incident; the next `relink_unit` fixes it — never restart a serving endpoint for cosmetics.
|
|
- Fixing the script = commit + `./scripts/deploy.sh <host> --service <svc>`; never edit on the host (a direct `scp` leaves drift that blocks the next deploy's ff-only pull with "local changes would be overwritten"; repair with `git checkout -- <file>`).
|