Frank
dab483c79f
Engine: digest-pinned docker.io/kyuz0/vllm-therock-gfx1151 (vLLM 0.30, TheRock ROCm 10, Triton GDN kernels); the serve script gates the image by REPO DIGEST, the payload (config + index + exact shard count), the spec draft (BOTH weight layouts: single model.safetensors for dflash, sharded index for the Qwen3.5-9B MTP head), and whole /dev/dri, sweeps the bench identity vllm-mimo, then exec podman wait so a container death trips Restart=always. Shape: 4 x 262144 at util 0.45 (KV grows only on the 8 full-attention layers), MTP 2 draft tokens. Measured 2026-10-01 window (frank-large stopped): base 13.1 t/s decode (~94% of the LPDDR5x BF16 roof), DFlash-8 17.4, MTP-2 19.8 with 71%/44% per-position acceptance - the cross-model head works because vLLM reads the draft from a separate path while MiMo ships zero mtp.* tensors. Spec arms do not preserve greedy bf16 text (batched-verify reduction order). The seam was rewritten around the new laws (18 tests): unit self-healing, digest gating, engine wiring, the measured spec table, the retired gufo gates, the deployer MainPID check. frank_small port test follows the PORT -> FRANK_LARGE_PUBLISH rename. Rollback: tag frank-large-gufo-q4kxl-20261001.
47 lines
2.1 KiB
Desktop File
47 lines
2.1 KiB
Desktop File
# frank-large.service — vLLM MiMo-V2.6-Distill-Qwen-9B (BF16) on the Strix
|
|
# Halo gfx1151 iGPU, :7770. 2026-10-01 cutover from gufo (tag
|
|
# frank-large-gufo-q4kxl-20261001 restores the Q4_K_XL + DFlash2 shape).
|
|
#
|
|
# WHY A ROOT UNIT: parity with frank-primary/frank-small — the unit is
|
|
# relink_unit'd from this repo into /etc/systemd/system/, PID1 grants
|
|
# LimitMEMLOCK=infinity (vLLM pins host buffers), and OOM protection lands
|
|
# on the engine instead of the co-tenants. The container runs in dank's
|
|
# ROOTLESS podman storage.
|
|
#
|
|
# The serve script drives `podman run` directly (image entrypoint is
|
|
# overridden to `vllm serve`) and blocks on the container, so a container
|
|
# death is a non-zero ExecStart. NOTE from frank-primary's experience:
|
|
# vLLM's API server can drain and exit 0 after an engine fault, so the
|
|
# restart policy MUST be always, not on-failure.
|
|
[Unit]
|
|
Description=frank-large — vLLM MiMo-V2.6-Distill-Qwen-9B (BF16) on the gfx1151 iGPU
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
# Never let the start-rate limiter give up after a crash loop.
|
|
StartLimitIntervalSec=0
|
|
|
|
[Service]
|
|
User=dank
|
|
Group=dank
|
|
LimitMEMLOCK=infinity
|
|
Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
|
|
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
|
|
# systemd stop: podman wait alone cannot stop the container child; stop it
|
|
# explicitly or the engine outlives the unit holding the GPU and port.
|
|
ExecStop=/usr/bin/podman stop -t 120 frank-large-mimo
|
|
# Boot cost: ~200-240 s warm (measured 2026-10-01 bench window); cold torch
|
|
# compile is longer. Health stays 503 until the engine is up — do not kill.
|
|
TimeoutStartSec=1800
|
|
# ExecStop allows 120 s of drain; the stop timeout must exceed it (the
|
|
# frank-primary 2026-10-01 lesson: a short TimeoutStopSec records 'Failed
|
|
# with result timeout' on every stop).
|
|
TimeoutStopSec=180
|
|
Restart=always
|
|
RestartSec=10
|
|
# The engine holds 17.5 GiB of weights plus the KV pool in the GTT
|
|
# aperture; it is the last process on this host the kernel should pick.
|
|
OOMScoreAdjust=-500
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|