Frank
d91379cff2
DFlash2 experiment declared failed 2026-10-01: the drafter serves and drafts (12.8 tok/s proposed) but acceptance is pinned at 1.0-1.38 on ALL text classes including verbatim repetition — the aux-hidden-state capture semantics are wrong (single-stream entering-state vs post-layer/comb is the untested alternative). At ~1.2 acceptance the 8-token verify tax makes decode 5x SLOWER than the 9.9-10.0 baseline (1.6-1.8 tok/s measured). Restores the verified 37-40 tok/s shape: qwen3.8-flash-next FP8, MTP4, prefetch off, 200k x 3 seqs, fp8 KV.
47 lines
2.2 KiB
Desktop File
47 lines
2.2 KiB
Desktop File
# frank-primary.service — tcclaviger RDNA4 vLLM endpoint (orcarouter
|
|
# Qwen3.8-Flash-Next-Uncensored-FP8, MTP4 spec decode, KV q8/q8, 600k total
|
|
# context over 3 sessions, TP=2, tiered experts, 131-shard payload) on the
|
|
# 2x R9700 pair, dankserver :7770. 2026-09-29 cutover from the R9V radiance
|
|
# fork (git tag frank-primary-r9v-q8-20260929 is the pre-cutover state; its
|
|
# launcher/shim/setup-state machinery is retired and the state dirs stay on
|
|
# disk as rollback).
|
|
#
|
|
# Same port, same served alias; the harness models.yml id is unchanged.
|
|
# The serve script drives `podman run` directly (the image entrypoint execs
|
|
# `vllm serve`) and then waits on the container, so a crashed engine is a
|
|
# nonzero ExecStart and Restart=on-failure is real.
|
|
#
|
|
# WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
|
|
# of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
|
|
# anonymous memory). Pinned pages are charged against RLIMIT_MEMLOCK; dank's
|
|
# login session on this host carries 8 MiB and a user quadlet is capped by the
|
|
# user manager's own hard limit — raising that means restarting
|
|
# user@1000.service and bouncing every user container on the host. A root unit
|
|
# grants LimitMEMLOCK=infinity to the dank process via PID1, and the image
|
|
# stays in dank's rootless podman storage.
|
|
[Unit]
|
|
Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
User=dank
|
|
Group=dank
|
|
# The tiered-expert offloader pins host RAM; an RLIMIT_MEMLOCK smaller than
|
|
# the pinned set fails the load mid-start. PID1 grants infinity here —
|
|
# parity with the fleet's other LLM units.
|
|
LimitMEMLOCK=infinity
|
|
Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env"
|
|
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
|
|
# systemd stop: wait alone cannot stop the container child; stop it
|
|
# explicitly. vLLM 0.26 drains gracefully on SIGTERM.
|
|
ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
|
|
# Boot cost: FP8 payload page-in plus torch compile/graph capture —
|
|
# ~15 min cold (measured 2026-09-29), faster warm.
|
|
TimeoutStartSec=1800
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|