Frank
ecaad36654
The llama.cpp b11030 trio becomes the vLLM 0.30 trio ahead of the FP8 cutover on :7770 — same port, same alias, no client moves. Serve shape: locally built ROCm gfx1201 image (localhost/frank-primary:vllm030) execs via podman as dank from the root unit (LimitMEMLOCK=infinity legalizes the pin). UVA offload: --offload-backend uva pins the PLE family (47.75 GiB, 138 tensors: 131 n-gram table shards + 7 per-layer conv/proj) + routed experts (114.86 GiB, 150,528 tensors) in host RAM — 162.60 GiB combined, ~81.3 GiB per rank under TP=2, hence --cpu-offload-gb 82 (80/card strands 2.6 GiB of experts on the cards). Parsers per the host's own Ornith-era precedent: qwen3_xml tools + qwen3 reasoning (vLLM 0.30 emits the channel as `reasoning` — reasoning_content is renamed server-side) + enable_thinking/preserve_thinking template kwargs. Spec decode stays env-wired and OFF at bring-up. Deployer deploy_llm_primary rewritten: image gate, index-driven shard-count gate (131), stale-container cleanup, port wait, and the false-green hardening — health is only green when the unit's own MainPID is live behind the answer (2026-09-26 lesson). Seam tests rewritten to the vLLM contract (15 gates): offload arithmetic, parser wiring, no-llama.cpp-ism, deployer gates. Suite 349/349 green.
38 lines
1.7 KiB
Desktop File
38 lines
1.7 KiB
Desktop File
# frank-primary.service — vLLM 0.30 endpoint (Qwen3.8-Flash-Next-FP8,
|
|
# UVA offload shape) on the 2x R9700 pair, dankserver :7770.
|
|
#
|
|
# Replaces the llama.cpp frank-primary unit of 2026-09-26 (same port, same
|
|
# served alias; harness models.yml unchanged). The serve script execs a
|
|
# podman container (image localhost/frank-primary:vllm030) as dank.
|
|
#
|
|
# WHY A ROOT UNIT (not a rootless quadlet): the UVA offload pins ~160 GiB of
|
|
# host RAM (PLE n-gram table + routed experts). Pinned pages are charged
|
|
# against RLIMIT_MEMLOCK; dank's login session on this host carries 8 MiB and
|
|
# a user quadlet is capped by the user manager's own hard limit — raising
|
|
# that means restarting user@1000.service and bouncing every user container
|
|
# on the host. A root unit grants LimitMEMLOCK=infinity to the dank process
|
|
# via PID1, no user-manager restart, and the image stays in dank's rootless
|
|
# podman storage (podman run as dank reads ~/.local/share/containers).
|
|
[Unit]
|
|
Description=frank-primary — vLLM 0.30 Qwen3.8-Flash-Next-FP8 (UVA offload) on 2x R9700
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
User=dank
|
|
Group=dank
|
|
# The UVA offloader pins the offloaded weights in host RAM; an RLIMIT_MEMLOCK
|
|
# smaller than the pinned set fails the load mid-start. PID1 grants infinity
|
|
# here — parity with the fleet's other LLM units.
|
|
LimitMEMLOCK=infinity
|
|
Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env"
|
|
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
|
|
# Cold start: Triton/inductor compile (several minutes on first boot) plus
|
|
# reading+pinning ~173 GiB of weights. Parity with the radiance-era timeout.
|
|
TimeoutStartSec=1800
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|