frank-primary — R9V radiance-vLLM Q8_0 (dankserver, ACTIVE)

The Dankserver LLM endpoint is the R9V radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned prebuilt image bundle 2dac17a2) serving the unsloth Qwen3.8-Flash-Next-GGUF Q8_0 payload (175.30 GiB, 6 GGUF shards, sha256-verified per part) with the blockfp8 MTP4 head and the mmproj, on both R9700s (gfx1201, BDFs 0000:03:00.0 / 0000:07:00.0) as frank-primary on :7770.

Cut over from the uncensored IQ4_XS payload on 2026-09-27 at Frank's direction. Rollback target: frank-primary-r9v-radiance-qwen38-uncensored — the uncensored state dir (/home/dank/r9v-state/uncensored/, ready: true), its frozen config (restore-config-20260927.env) and payload stay on disk; a rollback is an env-file R9V_STATE_FILE flip plus a redeploy.

Why the swap

The uncensored IQ4_XS payload measured 59–96 tok/s decode with draft-acceptance swings; the Q8_0 payload trades VRAM-resident expert headroom for precision — quantization to Q8_0 doubles expert bytes per block, so the tiered-expert residency budget halves (see live-llm-benchmark-methodology caveats on decode-vs-residency). The probe-era numbers are not the cutover numbers; see "Known costs" below.

Production shape (repo-tracked)

The repo carries a thin wiring; the engine shape lives on-host:

Piece	File	Law
Unit	`scripts/frank-primary.service`	Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-primary-r9v`, `Restart=on-failure`
Env	`scripts/frank-primary.env`	Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary`, container `frank-primary-r9v`, GPU BDFs, dev-overlay adapter, `R9V_FULL_MUTABLE_CACHE=0`, sharp chat template
Serve	`scripts/serve-frank-primary.sh`	Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait`
Deployer	`scripts/deploy-remote.sh` `deploy_llm_primary`	Gates on setup state ready, image digest, PLE bytes, all six shard files + mmproj; MainPID false-green check

Why the tuned shape lives on-host: the q8 profile's setup run qualified the R9V_* config (tiered-expert placement, per-rank offload budgets, 80-slot tiered cache, PLE pinned residency 54.4 GB, MTP4, parser wiring, MAX_MODEL_LEN=131072). It is written to /home/dank/r9v-state/q8xl/setup.json and the serve script layers it UNDER the repo env (repo keys win). Copying those values into the repo would drift from the qualification — don't.

Spec decode: MTP4 only — the ngram-mod ask is NOT expressible

llama.cpp chains drafter types (--spec-type draft-mtp,ngram-mod on frank-small/large). The fork's speculative_config is a single dict (one method per engine): mtp, ngram, ngram_gpu, dflash, … are mutually exclusive — no chaining seam exists in the fork. What IS active:

MTP4 (R9V_MTP_SPEC_TOKENS=4, blockfp8 draft head, draft TP=2) — measured mean acceptance length 3.19–5.00 during qualification.
The PLE n-gram — architectural (ngram_embedding module), rides along automatically; it is not the llama.cpp ngram-mod matcher and cannot be "added" as a second spec method.
ngram_gpu exists as an ALTERNATIVE single method (fork's GPU n-gram proposer) — swapping MTP4 for it is an either/or, not a chain.

Sharp chat template (fleet parity with frank-large)

scripts/qwen-sharp-v22.4.0.jinja is served via the launcher's opt-in R9V_CHAT_TEMPLATE_FILE passthrough (fork commit 30d4a32): the file is mounted read-only at /opt/r9v/share/chat-template.jinja and passed as --chat-template, which wins over the tokenizer dir's metadata/chat_template.jinja; R9V_CHAT_TEMPLATE_KWARGS carries --default-chat-template-kwargs {"preserve_thinking":true} (the template defaults preserve_thinking true natively; request-level kwargs still win per request). The template folds reasoning efforts (high/xhigh/max → xhigh, minimal/low → low), matching models.yml's reasoningEffortMap. The sha-pinned metadata/chat_template.jinja in the package is NEVER edited in place — mutating it drifts the qualification.

Two q8-only runtime laws

Quantized inject shard — the Q8_0 payload quantizes BOTH fused hyper-connection shards (the IQ4_XS package ships inject as F32), so the shard-uniformity check rejects the fused projection at load. R9V_DEV_QWEN4_EXP_ADAPTER_PY (fork commit 363d606) mounts a dev-overlay adapter that dequantizes a quantized inject shard in-stream; uncensored payloads pass through unchanged.
R9V_FULL_MUTABLE_CACHE=0 — the runtime overlay's env.always injects the uncensored 160-slot full-mutable cache geometry, which the q8 profile's 80-slot tiered cache fails with a baseline mismatch. The launcher passes this key AFTER the overlay envs (docker last-wins).

Fork checkout + shim (on-host, outside the repo)

Checkout: /home/dank/GitPackages/radiance-vllm-qwen4exp (fork commits 363d606 dev-overlay, 2fe6dbb, 50a79ec qualification doc, 30d4a32 chat-template passthrough — local-only, remote is GitHub)
State: /home/dank/r9v-state/q8xl/setup.json (0600, ready: true)
Model dir: /home/dank/models/Qwen3.8-Flash-Next-Q8_0-R9V/ — 175.30 GiB payload as six Q8_0/Qwen3.8-Flash-Next-Q8_0-0000N-of-00006.gguf shards
- mmproj; PLE derived at the pinned size 54,400,261,120 B
Docker→podman shim: /home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker (PATH-prepended by the serve script). The R9V tooling is docker-pinned; the shim carries three laws:
1. Image ID law — the bundle is a containerd/OCI docker-save; docker keys the image by its OCI MANIFEST digest (sha256:2dac17a2…), podman by config digest but preserves the manifest digest in RepoDigests. The shim answers image inspect sha256:X from RepoDigests and rewrites bare sha256:X run refs to repo@sha256:X.
2. LogConfig law — podman inspect answers HostConfig.LogConfig.Config: null where docker returns the log-opts map (the doctor crashes on .get("Config", {}) == null). The shim records run --log-opt pairs and normalizes inspect output.
3. docker info law — podman lacks SecurityOptions/ServerVersion; the shim answers rootless + 27.5.1 (liveness probe).
venv: ~/.r9v-venv (huggingface_hub for fetch/verify)

Bring-up battery (qualification, 2026-09-27)

Setup ready: true; all 20 payload artifacts size+sha256 verified (175.30 GiB total across the six shards).
Vision: 64×64 red PNG → "Red" (79 tok, 7.4 s end-to-end on the scratch port). Prose: 100 tok in 9.1 s ≈ 11.0 tok/s wall on a cold cache with reasoning tokens included in the wall clock — decode-rate context, not a bench (no throughput bench was run, per Frank's direction).
MTP4: mean acceptance length 3.19–5.00 across the battery.
PLE pinned proof: mapping_bytes = mapping_rss_bytes = 54,400,261,120 (fully RSS-resident); doubling for re-pin would need 108,800,522,240 ≤ available 140,876,808,192; registration took 446 s.

Memory shape (measured)

Load: 21.84 GiB / worker at ~158 s; cold start → serve ~16 min.
PLE pinned residency 54.4 GB in host RAM (vs the uncensored payload's ssd-mmap residency).
Context envelope: R9V_MAX_MODEL_LEN=131072; harness models.yml carries contextWindow: 131072, maxTokens: 36864 (compaction-gap math: 70% × 131,072 = 91,750 + 36,864 = 128,614 ≤ 131,072).
Prefix caching OFF (R9V_ENABLE_PREFIX_CACHING=0), CACHE192 off — the KV envelope at 131,072 is smaller than the uncensored 196,608 shape.

Ops

Restart: sudo systemctl restart frank-primary (root unit; the serve script re-gates everything, relaunch through the shim, waits).
First start after boot re-warms ~175 GiB of page cache; expect minutes to /health; TimeoutStartSec=1800.
Scratch/probe: NEVER probe against the prod state dir. Copy it, set R9V_HOST_PORT to a free port and R9V_CONTAINER_NAME to a scratch name, and stop the probe before any deploy — a probe holding :7770 produces a false-green deploy (the 2026-09-26 lesson, guarded by the MainPID check).
Rollback to uncensored: point scripts/frank-primary.env R9V_STATE_FILE back at /home/dank/r9v-state/uncensored/setup.json, drop the two q8-only keys (adapter, FULL_MUTABLE_CACHE) and the template keys, redeploy — the uncensored state dir and payload are intact on disk.
Models on the wire: served alias frank-primary; thinking channel is reasoning (vLLM); tool parser qwen3_coder.

Known costs (accepted at cutover)

Residency-halved experts — Q8_0 doubles expert bytes vs IQ4_XS, so the tiered cache holds half as many experts in VRAM at the same budget; the probe's 11 tok/s decode was residency-bound, and the production number depends on the re-qualified budgets in the q8 state.
128K envelope — down from the uncensored 196,608 (which itself was down from 262,144 on llama.cpp).
Reference quant — back to the unsloth Q8_0 weights (no abliteration); answer-quality profile differs from the uncensored incumbent by design.
Docker-pinned fork — updates flow through image bundles + the shim, never a bare container run; the launcher stays the single argv owner.

9.6 KiB Raw Blame History Unescape Escape