frank-medium: llama.cpp layer split across both R9700s, DFlash2 drafting
The vllm-radiance replica pair could not use both cards for one model: TP=2
needs a cross-card collective and the second R9700 cannot dispatch one. The
same wall blocks vLLM's pipeline parallelism, whose stage handoff is an RCCL
send/recv of the hidden state (distributed/parallel_state.py) -- a fact worth
writing down, because the obvious reading of the ask ('use PP to dodge the
atomics problem') is exactly backwards.
llama.cpp's layer split needs no collective at all: layers are distributed and
the boundary activation is a per-backend copy, once per forward pass. One
process, both cards, one model. The runtime is mainline llama.cpp at fb27a525
(the ref is pinned by DFlash2 AND qwen35 MTP; the ROCmFPX fork this host had
before has neither), built for gfx1201 against the private ROCm 7.14 toolchain
the earlier build already used, and bundled next to the binary.
Server: ajgazin Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the model's MTP
head in the file, plus the z-lab DFlash2 Q8_0 block-diffusion drafter at depth
7 (its block is 8, so 7 is the ceiling llama.cpp allows). Text only: vision
stays on frank-small.
Retired into the trash rather than deleted: the three vLLM quadlets, the
radiance entrypoint/healthcheck, the routing proxy and its tests, and the two
bench scripts that drove 'vllm bench serve' inside the retired image. The
deploy handler now stops them, waits for 7770 to be released, installs the
ROOT unit and refuses to start anything when the runtime or payload is
missing. The planner maps the scripts/ trio to frank-medium only, and the
scoping seam asserts the retired service dir is gone rather than merely
unmapped.
This commit is contained in:
Frank committed
1 parent
49e8dfff66
commit
05c0d3a00a
21 files changed
+608
-72
No files matched your search
@@ -1,98 +0,0 @@
|
||||
# Quadlet unit: frank-medium replica A (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankserver: ~/.config/containers/systemd/frank-medium-a.container
|
||||
# Unit name: frank-medium-a.service (container name: frank-medium-a)
|
||||
#
|
||||
# WHY TWO REPLICAS INSTEAD OF TP=2. The intended shape for frank-medium is one
|
||||
# tensor-parallel process across both R9700s (see frank-medium.container, kept
|
||||
# as the target). That shape cannot run on this host: RCCL's collective kernels
|
||||
# carry a hidden hostcall-buffer parameter, and the ROCr runtime refuses to
|
||||
# dispatch them on a device without PCIe atomics ("Pcie atomics not enabled,
|
||||
# hostcall not supported" -> AQL dispatch failed -> hipErrorIllegalState). The
|
||||
# second R9700 sits behind the chipset bridges 05:00.0/06:09.0, which advertise
|
||||
# `AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
|
||||
# atomics on that path; the amdgpu driver's pci_enable_atomic_ops_to_root()
|
||||
# therefore fails for that GPU and KFD marks the CPU<->GPU2 link NO_ATOMICS.
|
||||
# The first R9700 sits on CPU root port 00:01.1 (Routing+) and does have
|
||||
# atomics. Result: every collective that touches GPU 2 dies at launch.
|
||||
# Two independent single-GPU replicas use both cards instead; the proxy unit
|
||||
# keeps the single :7770 endpoint. Full evidence: the frank-medium runbook.
|
||||
#
|
||||
# SHAPE. vLLM binds 0.0.0.0:7771 on the host network directly; no PublishPort.
|
||||
# MAXLEN is 114688, not the TP=2 deployment's 262144: one card carries the
|
||||
# whole model (~20 GiB of weights + drafter), leaving ~5.2 GiB of KV, and
|
||||
# 131072 does not fit (measured: needs 5.58 GiB). KV_MEM=0 lets vLLM profile
|
||||
# this shape instead of using the 2x7551-32624 pin, which is only valid at
|
||||
# TP=2/262144.
|
||||
#
|
||||
# DEVICES. /dev/kfd + /dev/dri; HIP_VISIBLE_DEVICES=0 (the first R9700). The
|
||||
# 2 GiB Raphael iGPU enumerates as index 2 and is excluded by the pin.
|
||||
# Only HIP_VISIBLE_DEVICES is set, never ROCR_VISIBLE_DEVICES: ROCR filters
|
||||
# first and HIP indexes into the filtered list, so a single-valued pair
|
||||
# (ROCR=1,HIP=1) leaves an empty device set.
|
||||
#
|
||||
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
|
||||
# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
||||
[Unit]
|
||||
Description=frank-medium replica A — vllm-radiance MXFP4 Qwen3.8-27B on R9700 #1
|
||||
After=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=frank-medium-a
|
||||
Image=localhost/radiance-vllm:0.9.3
|
||||
AddDevice=/dev/kfd
|
||||
AddDevice=/dev/dri
|
||||
GroupAdd=keep-groups
|
||||
# The upstream launcher's exact podman posture (rootless).
|
||||
PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host
|
||||
Environment=HIP_VISIBLE_DEVICES=0
|
||||
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
|
||||
Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
|
||||
Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja
|
||||
Environment=SERVED_NAMES=frank-medium
|
||||
Environment=PORT=7771
|
||||
Environment=TP=1
|
||||
Environment=MAXSEQS=4
|
||||
Environment=CHUNK=8192
|
||||
Environment=MAXLEN=114688
|
||||
Environment=GPU_UTIL=0.98
|
||||
Environment=KV_MEM=0
|
||||
Environment=SPEC_METHOD=dflash
|
||||
Environment=SPEC=7
|
||||
# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards).
|
||||
Volume=/home/dank/models:/models
|
||||
# Tokenizer files ride the HF cache; HF_HUB_OFFLINE=1 is set in the entrypoint.
|
||||
Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface
|
||||
# Compile cache. Shared with the other replica and with the TP=2 target: the
|
||||
# entries are content-addressed, so shapes simply add subdirectories.
|
||||
Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache
|
||||
# The pinned radiance clone: patches applied at every container start.
|
||||
Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z
|
||||
# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The image's
|
||||
# shipped r4d.so predates the GDN overflow fix and NaNs this model.
|
||||
Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z
|
||||
# Entrypoint + healthcheck from this checkout, mounted read-only.
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
|
||||
# The entrypoint must be the container command; the image's own entrypoint
|
||||
# (which the patched script execs at the end) must not run unpatched.
|
||||
PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh
|
||||
# 200 only once the engine serves; during first-start compilation the
|
||||
# container is legitimately unhealthy. HealthCmd is split on spaces into an
|
||||
# argv array by the podman 5.x v1 generator, hence the file-based probe.
|
||||
HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
|
||||
HealthInterval=60s
|
||||
HealthTimeout=30s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=1800s
|
||||
Pull=never
|
||||
|
||||
[Service]
|
||||
# First start compiles Triton/inductor kernels (several minutes) before the
|
||||
# engine loads ~20 GiB of weights.
|
||||
TimeoutStartSec=1800
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -1,98 +0,0 @@
|
||||
# Quadlet unit: frank-medium replica B (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankserver: ~/.config/containers/systemd/frank-medium-b.container
|
||||
# Unit name: frank-medium-b.service (container name: frank-medium-a)
|
||||
#
|
||||
# WHY TWO REPLICAS INSTEAD OF TP=2. The intended shape for frank-medium is one
|
||||
# tensor-parallel process across both R9700s (see frank-medium.container, kept
|
||||
# as the target). That shape cannot run on this host: RCCL's collective kernels
|
||||
# carry a hidden hostcall-buffer parameter, and the ROCr runtime refuses to
|
||||
# dispatch them on a device without PCIe atomics ("Pcie atomics not enabled,
|
||||
# hostcall not supported" -> AQL dispatch failed -> hipErrorIllegalState). The
|
||||
# second R9700 sits behind the chipset bridges 05:00.0/06:09.0, which advertise
|
||||
# `AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
|
||||
# atomics on that path; the amdgpu driver's pci_enable_atomic_ops_to_root()
|
||||
# therefore fails for that GPU and KFD marks the CPU<->GPU2 link NO_ATOMICS.
|
||||
# The first R9700 sits on CPU root port 00:01.1 (Routing+) and does have
|
||||
# atomics. Result: every collective that touches GPU 2 dies at launch.
|
||||
# Two independent single-GPU replicas use both cards instead; the proxy unit
|
||||
# keeps the single :7770 endpoint. Full evidence: the frank-medium runbook.
|
||||
#
|
||||
# SHAPE. vLLM binds 0.0.0.0:7772 on the host network directly; no PublishPort.
|
||||
# MAXLEN is 114688, not the TP=2 deployment's 262144: one card carries the
|
||||
# whole model (~20 GiB of weights + drafter), leaving ~5.2 GiB of KV, and
|
||||
# 131072 does not fit (measured: needs 5.58 GiB). KV_MEM=0 lets vLLM profile
|
||||
# this shape instead of using the 2x7551-32624 pin, which is only valid at
|
||||
# TP=2/262144.
|
||||
#
|
||||
# DEVICES. /dev/kfd + /dev/dri; HIP_VISIBLE_DEVICES=1 (the second R9700). The
|
||||
# 2 GiB Raphael iGPU enumerates as index 2 and is excluded by the pin.
|
||||
# Only HIP_VISIBLE_DEVICES is set, never ROCR_VISIBLE_DEVICES: ROCR filters
|
||||
# first and HIP indexes into the filtered list, so a single-valued pair
|
||||
# (ROCR=1,HIP=1) leaves an empty device set.
|
||||
#
|
||||
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
|
||||
# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
||||
[Unit]
|
||||
Description=frank-medium replica B — vllm-radiance MXFP4 Qwen3.8-27B on R9700 #2
|
||||
After=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=frank-medium-b
|
||||
Image=localhost/radiance-vllm:0.9.3
|
||||
AddDevice=/dev/kfd
|
||||
AddDevice=/dev/dri
|
||||
GroupAdd=keep-groups
|
||||
# The upstream launcher's exact podman posture (rootless).
|
||||
PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host
|
||||
Environment=HIP_VISIBLE_DEVICES=1
|
||||
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
|
||||
Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
|
||||
Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja
|
||||
Environment=SERVED_NAMES=frank-medium
|
||||
Environment=PORT=7772
|
||||
Environment=TP=1
|
||||
Environment=MAXSEQS=4
|
||||
Environment=CHUNK=8192
|
||||
Environment=MAXLEN=114688
|
||||
Environment=GPU_UTIL=0.98
|
||||
Environment=KV_MEM=0
|
||||
Environment=SPEC_METHOD=dflash
|
||||
Environment=SPEC=7
|
||||
# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards).
|
||||
Volume=/home/dank/models:/models
|
||||
# Tokenizer files ride the HF cache; HF_HUB_OFFLINE=1 is set in the entrypoint.
|
||||
Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface
|
||||
# Compile cache. Shared with the other replica and with the TP=2 target: the
|
||||
# entries are content-addressed, so shapes simply add subdirectories.
|
||||
Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache
|
||||
# The pinned radiance clone: patches applied at every container start.
|
||||
Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z
|
||||
# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The image's
|
||||
# shipped r4d.so predates the GDN overflow fix and NaNs this model.
|
||||
Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z
|
||||
# Entrypoint + healthcheck from this checkout, mounted read-only.
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
|
||||
# The entrypoint must be the container command; the image's own entrypoint
|
||||
# (which the patched script execs at the end) must not run unpatched.
|
||||
PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh
|
||||
# 200 only once the engine serves; during first-start compilation the
|
||||
# container is legitimately unhealthy. HealthCmd is split on spaces into an
|
||||
# argv array by the podman 5.x v1 generator, hence the file-based probe.
|
||||
HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
|
||||
HealthInterval=60s
|
||||
HealthTimeout=30s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=1800s
|
||||
Pull=never
|
||||
|
||||
[Service]
|
||||
# First start compiles Triton/inductor kernels (several minutes) before the
|
||||
# engine loads ~20 GiB of weights.
|
||||
TimeoutStartSec=1800
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -1,61 +0,0 @@
|
||||
# Quadlet unit: frank-medium proxy (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankserver: ~/.config/containers/systemd/frank-medium-proxy.container
|
||||
# Unit name: frank-medium-proxy.service (container name: frank-medium-proxy)
|
||||
#
|
||||
# WHAT IT IS. A prefix-affinity reverse proxy in front of the two single-GPU
|
||||
# frank-medium replicas (:7771 on R9700 #1, :7772 on R9700 #2). It preserves
|
||||
# the fleet contract: one endpoint on host port 7770 answering as the served
|
||||
# model name frank-medium, OpenAI /v1, streaming included.
|
||||
#
|
||||
# WHY IT EXISTS. TP=2 — one vLLM process spanning both cards — is impossible
|
||||
# on this host (see frank-medium-a.container and the runbook: RCCL's kernels
|
||||
# need PCIe atomics, and the second card's chipset path advertises
|
||||
# AtomicOpsCap: Routing-). Two replicas are how both cards get used.
|
||||
#
|
||||
# ROUTING. Requests are keyed on a hash of their leading 4 KiB of prompt text,
|
||||
# so a conversation family — an agent loop re-sending the same system prompt,
|
||||
# tool list and file context every turn — keeps landing on the same replica,
|
||||
# where --enable-prefix-caching has that prefix warm. Plain round-robin threw
|
||||
# half of those cache hits away. A replica that is down is skipped, and
|
||||
# ROUTING=roundrobin restores the old behaviour for A/B measurement. /health
|
||||
# answers 200 while at least one replica is up and names the ones that are
|
||||
# down. The proxy runs on host networking so it can bind 7770 and reach the
|
||||
# replicas on loopback.
|
||||
#
|
||||
# The script is a few dozen lines of aiohttp, mounted read-only out of
|
||||
# /home/dank/serverAppConfigs (deploy.sh pulls that checkout before dispatch).
|
||||
# It reuses the same image as the replicas: aiohttp is already there and no
|
||||
# extra image has to be pulled on this host.
|
||||
[Unit]
|
||||
Description=frank-medium proxy — prefix-affinity in front of the two single-GPU replicas
|
||||
After=network-online.target
|
||||
Wants=frank-medium-a.service frank-medium-b.service
|
||||
After=frank-medium-a.service frank-medium-b.service
|
||||
|
||||
[Container]
|
||||
ContainerName=frank-medium-proxy
|
||||
Image=localhost/radiance-vllm:0.9.3
|
||||
PodmanArgs=--network=host
|
||||
Environment=PORT=7770
|
||||
Environment=UPSTREAMS=http://127.0.0.1:7771,http://127.0.0.1:7772
|
||||
# The proxy is two files: routing.py holds the replica-selection policy, the
|
||||
# server holds the HTTP plumbing. Both are mounted because the server imports
|
||||
# the policy by name from its own directory (/usr/local/bin is sys.path[0]).
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/proxy/routing.py:/usr/local/bin/routing.py:ro
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/proxy/affinity-proxy.py:/usr/local/bin/affinity-proxy.py:ro
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
|
||||
PodmanArgs=--entrypoint /usr/local/bin/affinity-proxy.py
|
||||
HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
|
||||
HealthInterval=60s
|
||||
HealthTimeout=30s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=60s
|
||||
Pull=never
|
||||
|
||||
[Service]
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -1,147 +0,0 @@
|
||||
# Quadlet unit: frank-medium (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankserver: ~/.config/containers/systemd/frank-medium.container
|
||||
# Unit name: frank-medium.service (container name: frank-medium)
|
||||
#
|
||||
# The dankserver LLM endpoint (:7770, alias frank-medium), since 2026-09-15
|
||||
# running the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3
|
||||
# image + the ggz14/radiance-vllm-mxfp4 patches) serving
|
||||
# just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on the 2x Radeon AI PRO
|
||||
# R9700 (gfx1201) cards. It replaced the charlie12345/ROCmFPX llama.cpp
|
||||
# deployment, which served the same alias from the Q8_0_ROCMFPX_AGENT GGUF;
|
||||
# the MXFP4 stack is tensor-parallel (TP=2, R4D one-shot all-reduce) with the
|
||||
# DFlash2-FP8 block-diffusion drafter (SPEC=7) — the measured production
|
||||
# configuration of the upstream repo, on our exact hardware signature
|
||||
# (2x7551-32624, see the KV pin below).
|
||||
#
|
||||
# SHAPE. The launcher the upstream repo ships binds the server with
|
||||
# --network=host (vLLM takes --host 0.0.0.0 --port 7770 directly), which is
|
||||
# what this unit does too — no PublishPort. The fleet contract is unchanged:
|
||||
# :7770, served model name frank-medium (SERVED_NAMES), OpenAI /v1.
|
||||
#
|
||||
# NOT INSTALLED ON THIS HOST. This is the target shape: one tensor-parallel
|
||||
# process across both R9700s. It cannot launch a collective here — RCCL's
|
||||
# kernels need PCIe atomics and the second card sits behind chipset bridges
|
||||
# advertising `AtomicOpsCap: Routing-` — so the active deployment is the
|
||||
# two-replica trio in this directory (frank-medium-a/b/proxy) and
|
||||
# deploy-remote.sh deliberately does not install this unit. Keep it for a
|
||||
# host whose second card is on an atomic-capable root port; the full evidence
|
||||
# is in serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md.
|
||||
#
|
||||
# DEVICES. /dev/kfd + /dev/dri through the amdgpu/KFD stack; HIP/ROCR device
|
||||
# indices 0,1 (the two R9700s). The 2 GiB Raphael iGPU enumerates as index 2
|
||||
# and is excluded both by the index pin and by the upstream MIN_GPU_MIB=8192
|
||||
# logic this unit replaces. --privileged --ipc=host --security-opt
|
||||
# seccomp=unconfined --cap-add SYS_PTRACE is the upstream launcher's exact
|
||||
# podman posture.
|
||||
#
|
||||
# MOUNTS. /models (rw checkpoints), the HF cache (tokenizer files),
|
||||
# /cache (Triton/inductor/aiter/vLLM compile cache — first start compiles for
|
||||
# several minutes, later starts reuse), /patches (the pinned radiance clone —
|
||||
# its patches are applied at EVERY container start by the entrypoint), /r4d
|
||||
# (the patched libr4d build; the image's shipped r4d.so NaNs the
|
||||
# gated-delta-net and must never be used), and the entrypoint + healthcheck
|
||||
# scripts from this checkout (mounted read-only straight out of
|
||||
# /home/dank/serverAppConfigs — deploy.sh git-pulls the checkout before
|
||||
# dispatching).
|
||||
#
|
||||
# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
|
||||
# path cannot resolve registry-prefixed refs from local storage at create
|
||||
# time (same as frank-large on dankclaw). deploy-remote.sh pulls the
|
||||
# docker.io ref and re-tags it before the restart. Pull=never keeps boots off
|
||||
# the registry.
|
||||
#
|
||||
# KEEP IN SYNC: the radiance clone is pinned (97353293...); if it is updated,
|
||||
# re-diff dankserver/frank-medium/radiance-entrypoint.sh against the new
|
||||
# serve-mxfp4.sh container command and bump the pin deliberately.
|
||||
#
|
||||
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
|
||||
# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
||||
[Unit]
|
||||
Description=frank-medium — vllm-radiance MXFP4 Qwen3.8-27B-Uncensored on 2x R9700 (gfx1201)
|
||||
After=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=frank-medium
|
||||
Image=localhost/radiance-vllm:0.9.3
|
||||
# vLLM binds 0.0.0.0:7770 on the host network directly; no PublishPort.
|
||||
AddDevice=/dev/kfd
|
||||
AddDevice=/dev/dri
|
||||
GroupAdd=keep-groups
|
||||
# The upstream launcher's exact podman posture (rootless; podman-only flags
|
||||
# like --replace are not needed under systemd management).
|
||||
PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host
|
||||
# The two R9700s. The Raphael iGPU (index 2, 2 GiB) stays out: shared system
|
||||
# memory is not a compute target.
|
||||
Environment=ROCR_VISIBLE_DEVICES=0,1
|
||||
Environment=HIP_VISIBLE_DEVICES=0,1
|
||||
# The uncensored MXFP4 checkpoint ships its fp8 MTP head already, so no
|
||||
# fp8_mtp.py rewrite step exists or is needed (unlike amd/Qwen3.8-27B-Quark-AWQ-MXFP4).
|
||||
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
|
||||
Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
|
||||
Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja
|
||||
# Fleet identity: the id at /v1/models and in every response body. Every
|
||||
# client (Hermes, the harness, services.json) points at frank-medium.
|
||||
Environment=SERVED_NAMES=frank-medium
|
||||
Environment=PORT=7770
|
||||
# Fixed 2-card tensor parallel: the tested configuration of the upstream
|
||||
# repo; the R4D one-shot all-reduce kernel serves it natively.
|
||||
Environment=TP=2
|
||||
# Measured production batch shape (upstream README): MAXSEQS 8 peaks
|
||||
# aggregate throughput at 573 t/s; MAXLEN is the model's native context.
|
||||
Environment=MAXSEQS=8
|
||||
Environment=CHUNK=8192
|
||||
Environment=MAXLEN=262144
|
||||
Environment=GPU_UTIL=0.98
|
||||
# Speculative decoding: the DFlash2 drafter at depth 7 is the configuration
|
||||
# every published number (224 t/s single-stream decode, 4749-4810 t/s
|
||||
# prefill) was measured with.
|
||||
Environment=SPEC_METHOD=dflash
|
||||
Environment=SPEC=7
|
||||
# KV cache pin, bytes. From the upstream repo's shipped kv-profiles.tsv row
|
||||
# sig=2x7551-32624 maxseqs=8 chunk=8192 maxlen=262144 spec=dflash
|
||||
# which is EXACTLY this host (2x R9700 DID 0x7551, 32624 MiB each). Pinned
|
||||
# explicitly because this unit does not run the launcher's lookup; valid
|
||||
# only at this batch shape (change MAXSEQS/CHUNK/SPEC_METHOD => recalibrate
|
||||
# with calibrate-kv.sh or set KV_MEM=0 to let vLLM profile). Worth
|
||||
# 943,581 KV tokens vs ~892,799 profiled (~5.7%).
|
||||
Environment=KV_MEM=18563072000
|
||||
# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards;
|
||||
# keep parity with the launcher's rw mount).
|
||||
Volume=/home/dank/models:/models
|
||||
# Tokenizer files ride the HF cache; HF_HUB_OFFLINE=1 is set in the
|
||||
# entrypoint.
|
||||
Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface
|
||||
# Compile cache. Keyed on model + stack version + every graph-changing knob;
|
||||
# never share it across configurations (upstream rule).
|
||||
Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache
|
||||
# The pinned radiance clone: patches applied at every container start.
|
||||
Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z
|
||||
# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The
|
||||
# image's shipped r4d.so predates the GDN overflow fix and NaNs this model.
|
||||
Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z
|
||||
# Entrypoint + healthcheck from this checkout, mounted read-only.
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
|
||||
# The entrypoint must be the container command; the image's own entrypoint
|
||||
# (which the patched script execs at the end) must not run unpatched.
|
||||
PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh
|
||||
# 200 only once the engine serves; during first-start compilation the
|
||||
# container is legitimately unhealthy. HealthCmd is split on spaces into an
|
||||
# argv array by the podman 5.x v1 generator, hence the file-based probe.
|
||||
HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
|
||||
HealthInterval=60s
|
||||
HealthTimeout=30s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=1800s
|
||||
Pull=never
|
||||
|
||||
[Service]
|
||||
# First start compiles Triton/inductor kernels (several minutes) before the
|
||||
# engine loads ~9.5 GiB of weights per card.
|
||||
TimeoutStartSec=1800
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
Reference in new issue
Block a user