frank-primary cutover: tcclaviger RDNA4 stack, 600k ctx / 3 sessions / q8-q8 KV
Replace the R9V radiance fork with the tcclaviger RDNA4 vLLM stack (the digest-pinned image the 2026-09-29 tuning campaign measured), serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (131 shards, 172.8 GiB, already on disk) with MTP4 spec decode. Qualified by boot probe on 2026-09-29: - MAXLEN=200000 x SEQS=3 (600k total) at --kv-cache-dtype fp8 (q8/q8 KV): 'GPU KV cache size: 729,545 tokens; Maximum concurrency for 200,000 tokens per request: 3.65x' at KV_BYTES 7.2 GiB PER RANK. - KV_BYTES is per-rank: global 16G and 9G both OOM (worker tried to allocate the full size; ~7.5 GiB/rank is the ceiling with the arena). - prefetch OFF by law (telemetry: 36% precision, ~50% wasted traffic); TunableOp off (corrupts numerics). - decode at this shape: 42.94/34.55/36.09 tok/s; arena 6,032 slots. Trio: frank-primary.env (all FRANK_PRIMARY_* keys), serve-frank-primary.sh (direct podman run; entrypoint execs vllm serve; waits on the container), frank-primary.service (root unit, same ExecStop law, new container name frank-primary-qwen). Deployer gates rewritten (image digest + shard count + weight-map index). download-frank-primary.sh is now inventory/verify (repo is GATED — 401 anonymous, masked LFS hashes; payload already on disk). Seam rewritten for the new contract. Pre-cutover tag: frank-primary-r9v-q8-20260929.
This commit is contained in:
1 parent
9cf4b1de82
commit
3bb0e82ccf
6 files changed
+409
-389
No files matched your search
@@ -1,24 +1,26 @@
|
||||
# frank-primary.service — R9V radiance-vLLM endpoint (unsloth
|
||||
# Qwen3.8-Flash-Next Q8_0, MTP4, TP=2, tiered experts, 6-shard payload) on
|
||||
# the 2x R9700 pair, dankserver :7770. 2026-09-27 cutover from the
|
||||
# uncensored IQ4_XS payload (its state dir stays on disk as rollback).
|
||||
# frank-primary.service — tcclaviger RDNA4 vLLM endpoint (orcarouter
|
||||
# Qwen3.8-Flash-Next-Uncensored-FP8, MTP4 spec decode, KV q8/q8, 600k total
|
||||
# context over 3 sessions, TP=2, tiered experts, 131-shard payload) on the
|
||||
# 2x R9700 pair, dankserver :7770. 2026-09-29 cutover from the R9V radiance
|
||||
# fork (git tag frank-primary-r9v-q8-20260929 is the pre-cutover state; its
|
||||
# launcher/shim/setup-state machinery is retired and the state dirs stay on
|
||||
# disk as rollback).
|
||||
#
|
||||
# Replaces the llama.cpp frank-primary unit (same port, same served alias;
|
||||
# harness models.yml unchanged in id, retuned for the 131072 window). The
|
||||
# serve script drives the R9V fork's own launcher through the docker->podman
|
||||
# shim and then waits on the container, so a crashed engine is a nonzero
|
||||
# ExecStart and Restart=on-failure is real.
|
||||
# Same port, same served alias; the harness models.yml id is unchanged.
|
||||
# The serve script drives `podman run` directly (the image entrypoint execs
|
||||
# `vllm serve`) and then waits on the container, so a crashed engine is a
|
||||
# nonzero ExecStart and Restart=on-failure is real.
|
||||
#
|
||||
# WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
|
||||
# of GiB of host RAM (the UVA expert host copy plus the 16 GiB PLE pinned
|
||||
# reserve). Pinned pages are charged against RLIMIT_MEMLOCK; dank's login
|
||||
# session on this host carries 8 MiB and a user quadlet is capped by the user
|
||||
# manager's own hard limit — raising that means restarting user@1000.service
|
||||
# and bouncing every user container on the host. A root unit grants
|
||||
# LimitMEMLOCK=infinity to the dank process via PID1, and the image stays in
|
||||
# dank's rootless podman storage (the shim drives podman as dank).
|
||||
# of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
|
||||
# anonymous memory). Pinned pages are charged against RLIMIT_MEMLOCK; dank's
|
||||
# login session on this host carries 8 MiB and a user quadlet is capped by the
|
||||
# user manager's own hard limit — raising that means restarting
|
||||
# user@1000.service and bouncing every user container on the host. A root unit
|
||||
# grants LimitMEMLOCK=infinity to the dank process via PID1, and the image
|
||||
# stays in dank's rootless podman storage.
|
||||
[Unit]
|
||||
Description=frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0 on 2x R9700
|
||||
Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
@@ -33,9 +35,9 @@ Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-pr
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
|
||||
# systemd stop: wait alone cannot stop the container child; stop it
|
||||
# explicitly. vLLM 0.26 drains gracefully on SIGTERM.
|
||||
ExecStop=/usr/bin/podman stop -t 120 frank-primary-r9v
|
||||
# Boot cost: the launcher's preflight doctor (~1 min) after a warm page
|
||||
# cache; cold (post-reboot) payload page-in of ~175 GiB dominates.
|
||||
ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
|
||||
# Boot cost: FP8 payload page-in plus torch compile/graph capture —
|
||||
# ~15 min cold (measured 2026-09-29), faster warm.
|
||||
TimeoutStartSec=1800
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
Reference in new issue
Block a user