Frank
c769ff2f63
Re-applies the desired shape incrementally on the proven-good fp8 3x262144
base, after bisecting why the first attempt (30904ba) crash-looped.
Root cause: `--max-num-seqs 2` cannot compile on this model. AOT
build_guards(strict_error=True) inside profile_run -> _dummy_run specializes
`query_start_loc`'s width to SEQS + 1 (3 at SEQS 2, 4 at SEQS 3) and the
ConstraintViolationError kills engine init ~7 min in. MNBT is orthogonal:
SEQS 2 failed at MNBT 1024 AND 2048; SEQS 3 passed at both. KV dtype was a
red herring. Measured by four single-variable boot probes.
Arena lever measured: the expert arena is sized from `max_model_len` via the
registry-priced KV reserve (9190 B/token x max_model_len), NOT from
--kv-cache-memory-bytes. 262144 -> 14.30 GiB / 6,248 slots; 200704 ->
14.87 GiB / 6,496 slots. The 6.1e9 -> 3.15e9 pool cut bought zero arena and
is a runtime-headroom lever.
Shape: MAXLEN 200704, SEQS 3 (unchanged - the only slot count that compiles),
KV_BYTES 3.15e9/rank = 2.04x full-length concurrency at 7,701 B/token fp8,
MNBT 1024, gather 32x64, no spec decode, no prefetch, no precise-memory.
68 lines
3.5 KiB
Desktop File
68 lines
3.5 KiB
Desktop File
# frank-primary.service — tcclaviger RDNA4 vLLM endpoint (orcarouter
|
|
# Qwen3.8-Flash-Next-Uncensored-FP8, fp8 KV, NO spec decode, 3 slots x 200704
|
|
# context with the pool sized to ~2 full-length conversations, TP=2, tiered
|
|
# experts, 131-shard payload) on the 2x R9700 pair, dankserver :7770.
|
|
# 2026-09-29 cutover from the R9V radiance fork (git tag
|
|
# frank-primary-r9v-q8-20260929 is the pre-cutover state; its
|
|
# launcher/shim/setup-state machinery is retired and the state dirs stay on
|
|
# disk as rollback).
|
|
#
|
|
# Same port, same served alias; the harness models.yml id is unchanged.
|
|
# The serve script drives `podman run` directly (the image entrypoint execs
|
|
# `vllm serve`) and then waits on the container. NOTE: that wait can return
|
|
# 0 even when the engine died mid-request (vLLM's API server drains and
|
|
# exits clean on a worker fault), which is why the restart policy is
|
|
# `always` — see the [Service] note.
|
|
#
|
|
# WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
|
|
# of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
|
|
# anonymous memory). Pinned pages are charged against RLIMIT_MEMLOCK; dank's
|
|
# login session on this host carries 8 MiB and a user quadlet is capped by the
|
|
# user manager's own hard limit — raising that means restarting
|
|
# user@1000.service and bouncing every user container on the host. A root unit
|
|
# grants LimitMEMLOCK=infinity to the dank process via PID1, and the image
|
|
# stays in dank's rootless podman storage.
|
|
[Unit]
|
|
Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
# Never let the start-rate limiter give up after a crash loop (a failed LLM
|
|
# endpoint that systemd refuses to retry is worse than a retrying one).
|
|
StartLimitIntervalSec=0
|
|
|
|
[Service]
|
|
User=dank
|
|
Group=dank
|
|
# The tiered-expert offloader pins host RAM; an RLIMIT_MEMLOCK smaller than
|
|
# the pinned set fails the load mid-start. PID1 grants infinity here —
|
|
# parity with the fleet's other LLM units.
|
|
LimitMEMLOCK=infinity
|
|
Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env"
|
|
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
|
|
# systemd stop: wait alone cannot stop the container child; stop it
|
|
# explicitly. vLLM 0.26 drains gracefully on SIGTERM.
|
|
ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
|
|
# Boot cost: FP8 payload page-in plus torch compile/graph capture —
|
|
# ~15 min cold (measured 2026-09-29), faster warm.
|
|
TimeoutStartSec=1800
|
|
# Restart=always, NOT on-failure (measured 2026-10-01): a runtime
|
|
# torch.OutOfMemoryError kills the engine while vLLM's API server still
|
|
# exits 0, so ExecStart returned clean, systemd logged "Deactivated
|
|
# successfully", and on-failure NEVER fired — the endpoint stayed down until
|
|
# a human noticed. `always` restarts on any container death; an explicit
|
|
# `systemctl stop` still wins (systemd does not restart a unit the operator
|
|
# stopped), so bench windows and deploys are unaffected.
|
|
Restart=always
|
|
RestartSec=10
|
|
# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without an
|
|
# explicit TimeoutStopSec systemd's 90 s default cuts the stop short, kills it,
|
|
# and records 'Failed with result timeout' (measured 2026-10-01 13:27:17).
|
|
# Must exceed the podman grace with margin for the engine's own shutdown.
|
|
TimeoutStopSec=180
|
|
# The engine pins tens of GiB of host RAM (expert host slabs); it is the
|
|
# last process on this host that should be chosen by the kernel OOM killer.
|
|
OOMScoreAdjust=-500
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|