Frank
1919ad7f31
Replace the FreeToken PR587 endpoint (single-GPU: its TP=2 dies in the pynccl.cu ROCm JIT) with the radiance engine on both R9700s, TP=2 via libr4d's fused one-shot all-reduce. Official Qwen/Qwen3.8-Flash-Next-FP8 payload at native block-FP8 precision: - scripts/build-frank-primary-rad.sh: staging (quantization_config drop = plugin SELECTION law) -> checkpoint reshape -> rad-convert with ONE --quant rule (PLE rtn fixed-scale); plan-only first. - scripts/stack-qwen38-official-fp8.py: the two value-preserving reshapes (experts byte-stacked into the plugin's stacked layout; PLE's 128 scaleless fp8 shards decoded to one bf16 shard with the checkpoint's own scale; the rtn rule re-encodes at that same scale - codes round-trip exactly and land the one-element scale plane the plugin declares). - scripts/hfget.sh + verified download (131/131 shards sha256-match). - froggeric v22.5 chat template vendored (--override-chat-template). - serve script (pinned ledger: 2x40 GiB pools + 47.7 GiB PLE + prefix host), TTM preflight fixed for kernels with pages_limit only (no pages_max - an absent knob is skipped, not read as zero); pages_limit raised to 52,428,800 pages = 200 GiB on the host pre-cutover. - root unit, relinked; deployer gates image digest + .rad floor 158,000 MiB + health; prometheus re-adds the /metrics job. - rollback: tag frank-primary-freetoken-pr587-20261005 (FreeToken trio + Dockerfile); payload + image stay on disk until the window closes. deploy-remote.sh also carries the parallel session's frank-small payload gates (swift15) - committed here per Frank's direction to commit the tree.
65 lines
3.4 KiB
Desktop File
65 lines
3.4 KiB
Desktop File
# frank-primary.service — radiance (StillDeadcode, Apache-2.0) endpoint
|
|
# (OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 at NATIVE block-FP8 precision, the
|
|
# rad-convert container qwen4exp-fp8-native.rad) on BOTH R9700s (TP=2,
|
|
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (git tag
|
|
# frank-primary-freetoken-pr587-20261005 is the pre-cutover state; the
|
|
# retired unit/env/serve trio and Dockerfile are in that tag).
|
|
#
|
|
# Same port, same served alias, same container identity as the unit name;
|
|
# the harness models.yml id is unchanged. The serve script drives
|
|
# `podman run` directly: the digest-pinned stilldeadcode/radiance image
|
|
# (287 MiB, kernels prebuilt for gfx1201) has the engine as its
|
|
# entrypoint. The image lives in dank's ROOTLESS podman storage.
|
|
# WHY A ROOT UNIT (not a rootless quadlet): radiance pins ~132 GiB of HOST
|
|
# RAM (2 x 40 GiB expert pools + the 47.7 GiB PLE ngram table + the prefix
|
|
# host tier). Pinned pages are charged against RLIMIT_MEMLOCK; dank's login
|
|
# session on this host carries 8 MiB and a user quadlet is capped by the
|
|
# user manager's own hard limit — raising that means restarting
|
|
# user@1000.service and bouncing every user container on the host. A root
|
|
# unit grants LimitMEMLOCK=infinity to the dank process via PID1; the
|
|
# engine still runs out of dank's rootless podman storage.
|
|
[Unit]
|
|
Description=frank-primary — radiance Qwen3.8-Flash-Next-FP8 (native) on 2x R9700 TP=2
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
# Never let the start-rate limiter give up after a crash loop (a failed LLM
|
|
# endpoint that systemd refuses to retry is worse than a retrying one).
|
|
StartLimitIntervalSec=0
|
|
|
|
[Service]
|
|
User=dank
|
|
Group=dank
|
|
# The engine pins ~132 GiB of host RAM (expert pools + PLE table); an
|
|
# RLIMIT_MEMLOCK smaller than the pinned set fails the load mid-start.
|
|
# PID1 grants infinity here — parity with the fleet's other LLM units.
|
|
LimitMEMLOCK=infinity
|
|
Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env"
|
|
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
|
|
# systemd stop: wait alone cannot stop the container child; stop it
|
|
# explicitly. No graceful drain contract is claimed (the OpenAI front end
|
|
# holds no queue to drain — in-flight decodes are lost, which a stop
|
|
# during a bench window accepts; radiance takes SIGTERM on the engine
|
|
# process and podman reaps the container).
|
|
ExecStop=/usr/bin/podman stop -t 120 frank-primary
|
|
# Boot cost: .rad load (weights + PLE upload) + expert host-pool pin. The
|
|
# publisher's compose healthcheck allows a 15-minute start_period; the
|
|
# vLLM-era budget (1800 s) carried forward.
|
|
# Restart=always, NOT on-failure (the 2026-10-01 vLLM lesson carries over:
|
|
# a clean API-server exit after a worker fault reads as success to
|
|
# systemd). `always` restarts on any container death; an explicit
|
|
# `systemctl stop` still wins, so bench windows and deploys are unaffected.
|
|
Restart=always
|
|
RestartSec=10
|
|
# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without
|
|
# an explicit TimeoutStopSec systemd's 90 s default cuts the stop short,
|
|
# kills it, and records 'Failed with result timeout' (measured 2026-10-01
|
|
# 13:27:17). Must exceed the podman grace with margin.
|
|
TimeoutStopSec=180
|
|
# The engine pins tens of GiB of host RAM (expert rows + PLE table); it is
|
|
# the last process on this host that should be chosen by the kernel OOM
|
|
# killer.
|
|
OOMScoreAdjust=-500
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|