frank-primary: TimeoutStopSec=180, drop duplicate StartLimitIntervalSec
ExecStop runs 'podman stop -t 120' but the unit never set TimeoutStopSec, so systemd's 90 s default cut the graceful stop short: journal shows 'Stopping timed out. Terminating.' then 'Failed with result timeout' on 2026-10-01 13:27:17 — every stop of this unit was logging a failed stop. 180 s covers the engine's drain with margin, and the seam now guards the RELATIONSHIP (TimeoutStopSec > the podman grace) rather than a literal. StartLimitIntervalSec is a [Unit] directive; the copy in [Service] made systemd log 'Unknown key StartLimitIntervalSec in section Service' on every load. Seam asserts it appears exactly once and not in [Service].
This commit is contained in:
1 parent
a1b650aabc
commit
e3c3fad0f1
2 files changed
+16
-2
No files matched your search
@@ -53,8 +53,11 @@ TimeoutStartSec=1800
|
|||||||
# stopped), so bench windows and deploys are unaffected.
|
# stopped), so bench windows and deploys are unaffected.
|
||||||
Restart=always
|
Restart=always
|
||||||
RestartSec=10
|
RestartSec=10
|
||||||
# Do not let the start-rate limiter give up after a crash loop.
|
# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without an
|
||||||
StartLimitIntervalSec=0
|
# explicit TimeoutStopSec systemd's 90 s default cuts the stop short, kills it,
|
||||||
|
# and records 'Failed with result timeout' (measured 2026-10-01 13:27:17).
|
||||||
|
# Must exceed the podman grace with margin for the engine's own shutdown.
|
||||||
|
TimeoutStopSec=180
|
||||||
# The engine pins tens of GiB of host RAM (expert host slabs); it is the
|
# The engine pins tens of GiB of host RAM (expert host slabs); it is the
|
||||||
# last process on this host that should be chosen by the kernel OOM killer.
|
# last process on this host that should be chosen by the kernel OOM killer.
|
||||||
OOMScoreAdjust=-500
|
OOMScoreAdjust=-500
|
||||||
|
|||||||
@@ -47,6 +47,17 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
|||||||
# (measured 2026-10-01 — the endpoint stayed down until a human noticed).
|
# (measured 2026-10-01 — the endpoint stayed down until a human noticed).
|
||||||
grep -q '^Restart=always$' "$UNIT"
|
grep -q '^Restart=always$' "$UNIT"
|
||||||
! grep -q '^Restart=on-failure$' "$UNIT"
|
! grep -q '^Restart=on-failure$' "$UNIT"
|
||||||
|
# Exiting the engine takes as long as vLLM's drain: ExecStop runs
|
||||||
|
# `podman stop -t 120`, so the stop timeout MUST exceed 120 s or systemd
|
||||||
|
# kills the stop and records 'Failed with result timeout' (measured
|
||||||
|
# 2026-10-01). Guard the relationship, not just the literal.
|
||||||
|
stop_t=$(awk -F= '/^TimeoutStopSec=/{print $2}' "$UNIT")
|
||||||
|
[ -n "$stop_t" ] && [ "$stop_t" -gt 120 ]
|
||||||
|
grep -q '^ExecStop=.*podman stop -t 120' "$UNIT"
|
||||||
|
# StartLimitIntervalSec belongs to [Unit]; a copy in [Service] is an
|
||||||
|
# 'Unknown key' warning systemd logs on every load.
|
||||||
|
[ "$(grep -c '^StartLimitIntervalSec=' "$UNIT")" = "1" ]
|
||||||
|
awk '/^\[Service\]/{in_svc=1} /^StartLimitIntervalSec=/{if(in_svc) exit 1}' "$UNIT"
|
||||||
grep -q '^RestartSec=' "$UNIT"
|
grep -q '^RestartSec=' "$UNIT"
|
||||||
grep -q '^User=dank$' "$UNIT"
|
grep -q '^User=dank$' "$UNIT"
|
||||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||||
|
|||||||
Reference in new issue
Block a user