Frank
a34322aeb0
The managed-skill library was regenerated: 94 narrow per-incident skills
replaced by 25 broader ones (radiance-*, llamacpp-*, homepage-*, fleet/
endpoint ops). Git renders this as mass delete + mass add because whole
directories changed name; old content stays in history at 12c5e88.
Surface check: the 25 on disk are exactly the skills the harness advertises
this session.
2.2 KiB
2.2 KiB
name, description
| name | description |
|---|---|
| llamacpp-serve-unit-boot-loop | Diagnose a Dank-fleet LLM systemd unit that boot-loops: classify 203/EXEC vs model-load OOM vs payload gate, using journal signatures and nvidia-smi/rocm-smi. |
llama.cpp / fleet serve unit boot loop
A fleet LLM unit (frank-large, frank-secondary, frank-primary) restarts every RestartSec. Read the journal FIRST, act second:
ssh <host> 'sudo systemctl status <svc> --no-pager | head -12; sudo journalctl -u <svc> --since "-6 min" -o cat --no-pager | tail -40'
Classify by signature
| Signature | Cause | Fix |
|---|---|---|
status=203/EXEC + Failed at step EXEC spawning + Mem peak: ~1.9M + GPU flat at 1 MiB |
the ExecStart script never ran: missing shebang, lost exec bit, or path moved by the deploy (git pull mid-restart) |
head -1 <script> must be #!/usr/bin/env bash; git ls-files -s <script> must show 100755; commit the fix, redeploy |
status=203/EXEC right after a deploy, then self-heals |
git replaced the file mid-checkout while the restart loop ran | ignore if the newest attempts succeed; verify podman ps --format '{{.CreatedAt}}' identity |
| Engine prints weights loading, then dies | VRAM/disk budget, engine flags | read the engine's own ledger (verbosity 4) for KV/compute-buffer MiB; price from two boots, never a formula |
| Health 503, no error, minutes of silence | normal cold load (big PLE/mmap) | do not kill; the listener binds after the full load |
Rules
- Judge the newest container/attempt only:
podman ps -a --format '{{.ID}} {{.CreatedAt}}',journalctl --since; a mid-stop container still serves old logs. systemctl is-activeisfailedafter every intentional stop on foreground-podman units (exit 143) — cosmetic.- The unit's
Descriptionis the last-deployed text, not the repo's. A stale Description is not an incident; the nextrelink_unitfixes it — never restart a serving endpoint for cosmetics. - Fixing the script = commit +
./scripts/deploy.sh <host> --service <svc>; never edit on the host (a directscpleaves drift that blocks the next deploy's ff-only pull with "local changes would be overwritten"; repair withgit checkout -- <file>).