---
name: llamacpp-serve-unit-boot-loop
description: "Diagnose a Dank-fleet LLM systemd unit that boot-loops: classify 203/EXEC vs model-load OOM vs payload gate, using journal signatures and nvidia-smi/rocm-smi."
---

# llama.cpp / fleet serve unit boot loop

A fleet LLM unit (`frank-large`, `frank-secondary`, `frank-primary`) restarts every `RestartSec`. Read the journal FIRST, act second:

```
ssh <host> 'sudo systemctl status <svc> --no-pager | head -12; sudo journalctl -u <svc> --since "-6 min" -o cat --no-pager | tail -40'
```

## Classify by signature

| Signature | Cause | Fix |
|---|---|---|
| `status=203/EXEC` + `Failed at step EXEC spawning` + `Mem peak: ~1.9M` + GPU flat at 1 MiB | the `ExecStart` script never ran: missing shebang, lost exec bit, or path moved by the deploy (git pull mid-restart) | `head -1 <script>` must be `#!/usr/bin/env bash`; `git ls-files -s <script>` must show `100755`; commit the fix, redeploy |
| `status=203/EXEC` right after a deploy, then self-heals | git replaced the file mid-checkout while the restart loop ran | ignore if the newest attempts succeed; verify `podman ps --format '{{.CreatedAt}}'` identity |
| Engine prints weights loading, then dies | VRAM/disk budget, engine flags | read the engine's own ledger (verbosity 4) for KV/compute-buffer MiB; price from two boots, never a formula |
| Health 503, no error, minutes of silence | normal cold load (big PLE/mmap) | do not kill; the listener binds after the full load |

## Rules
- Judge the newest container/attempt only: `podman ps -a --format '{{.ID}} {{.CreatedAt}}'`, `journalctl --since`; a mid-stop container still serves old logs.
- `systemctl is-active` is `failed` after every intentional stop on foreground-podman units (exit 143) — cosmetic.
- The unit's `Description` is the last-deployed text, not the repo's. A stale Description is not an incident; the next `relink_unit` fixes it — never restart a serving endpoint for cosmetics.
- Fixing the script = commit + `./scripts/deploy.sh <host> --service <svc>`; never edit on the host (a direct `scp` leaves drift that blocks the next deploy's ff-only pull with "local changes would be overwritten"; repair with `git checkout -- <file>`).