serverAppConfigs/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md

---
title: frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0
service: frank-primary
host: dankserver
port: 7770
status: active
deployed: 2026-09-27
supersedes: "[[frank-primary-r9v-radiance-qwen38-uncensored]]"
tags: [llm, frank-primary, r9v, vllm, radiance, gfx1201, q8]
---

# frank-primary — R9V radiance-vLLM Q8_0 (dankserver, ACTIVE)

The Dankserver LLM endpoint is the **R9V radiance-vLLM fork**
(`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt image bundle
`2dac17a2`) serving the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** payload
(175.30 GiB, 6 GGUF shards, sha256-verified per part) with the blockfp8
MTP4 head and the mmproj, on **both R9700s** (gfx1201, BDFs `0000:03:00.0`
/ `0000:07:00.0`) as `frank-primary` on **:7770**.

Cut over from the uncensored IQ4_XS payload on 2026-09-27 at Frank's
direction. Rollback target: [[frank-primary-r9v-radiance-qwen38-uncensored]]
— the uncensored state dir (`/home/dank/r9v-state/uncensored/`, `ready:
true`), its frozen config (`restore-config-20260927.env`) and payload stay
on disk; a rollback is an env-file `R9V_STATE_FILE` flip plus a redeploy.

## Why the swap

The uncensored IQ4_XS payload measured 59–96 tok/s decode with
draft-acceptance swings; the Q8_0 payload trades VRAM-resident expert
headroom for precision — quantization to Q8_0 doubles expert bytes per
block, so the tiered-expert residency budget halves — the probe-era
11 tok/s decode was residency-bound (expert streaming, not compute).
The "Known costs" section below carries the accepted trade.

## Production shape (repo-tracked)

The repo carries a **thin** wiring; the engine shape lives on-host:

| Piece | File | Law |
|---|---|---|
| Unit | `scripts/frank-primary.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-primary-r9v`, `Restart=on-failure` |
| Env | `scripts/frank-primary.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary`, container `frank-primary-r9v`, GPU BDFs, dev-overlay adapter, `R9V_FULL_MUTABLE_CACHE=0`, `R9V_PLE_READY_TIMEOUT=1800`, prefix caching ON, sharp chat template |
| Serve | `scripts/serve-frank-primary.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
| Deployer | `scripts/deploy-remote.sh` `deploy_llm_primary` | Gates on setup state ready, image digest, PLE bytes, all six shard files + mmproj; MainPID false-green check |

**Why the tuned shape lives on-host:** the q8 profile's setup run
qualified the `R9V_*` config (tiered-expert placement, per-rank offload
budgets, 80-slot tiered cache, PLE **pinned** residency 54.4 GB, MTP4,
parser wiring, `MAX_MODEL_LEN=131072`). It is written to
`/home/dank/r9v-state/q8xl/setup.json` and the serve script layers it
UNDER the repo env (repo keys win). Copying those values into the repo
would drift from the qualification — don't.

## Spec decode: MTP4 only — the ngram-mod ask is NOT expressible

llama.cpp chains drafter types (`--spec-type draft-mtp,ngram-mod` on
frank-small/large). The fork's `speculative_config` is a **single dict**
(one method per engine): `mtp`, `ngram`, `ngram_gpu`, `dflash`, … are
mutually exclusive — no chaining seam exists in the fork. What IS active:

- **MTP4** (`R9V_MTP_SPEC_TOKENS=4`, blockfp8 draft head, draft TP=2) —
  measured mean acceptance length 3.19–5.00 during qualification.
- **The PLE n-gram** — architectural (`ngram_embedding` module), rides
  along automatically; it is not the llama.cpp `ngram-mod` matcher and
  cannot be "added" as a second spec method.
- `ngram_gpu` exists as an ALTERNATIVE single method (fork's GPU n-gram
  proposer) — swapping MTP4 for it is an either/or, not a chain.

## Sharp chat template (fleet parity with frank-large)

`scripts/qwen-sharp-v22.4.0.jinja` is served via the launcher's opt-in
`R9V_CHAT_TEMPLATE_FILE` passthrough (fork commit `30d4a32`): the file is
mounted read-only at `/opt/r9v/share/chat-template.jinja` and passed as
`--chat-template`, which wins over the tokenizer dir's
`metadata/chat_template.jinja`; `R9V_CHAT_TEMPLATE_KWARGS` carries
`--default-chat-template-kwargs {"preserve_thinking":true}` (the template
defaults `preserve_thinking` true natively; request-level kwargs still win
per request). The template folds reasoning efforts (high/xhigh/max →
xhigh, minimal/low → low), matching `models.yml`'s `reasoningEffortMap`.
The sha-pinned `metadata/chat_template.jinja` in the package is NEVER
edited in place — mutating it drifts the qualification.

## Two q8-only runtime laws

1. **Quantized inject shard** — the Q8_0 payload quantizes BOTH fused
   hyper-connection shards (the IQ4_XS package ships inject as F32), so
   the shard-uniformity check rejects the fused projection at load.
   `R9V_DEV_QWEN4_EXP_ADAPTER_PY` (fork commit `363d606`) mounts a
   dev-overlay adapter that dequantizes a quantized inject shard
   in-stream; uncensored payloads pass through unchanged.
2. **`R9V_FULL_MUTABLE_CACHE=0`** — the runtime overlay's `env.always`
   injects the uncensored 160-slot full-mutable cache geometry, which the
   q8 profile's 80-slot tiered cache fails with a baseline mismatch. The
   launcher passes this key AFTER the overlay envs (docker last-wins).

## Fork checkout + shim (on-host, outside the repo)

- Checkout: `/home/dank/GitPackages/radiance-vllm-qwen4exp` (fork commits
  `363d606` dev-overlay, `2fe6dbb`, `50a79ec` qualification doc,
  `30d4a32` chat-template passthrough, `2de0357` PLE-timeout forwarding —
  local-only, remote is GitHub)
- State: `/home/dank/r9v-state/q8xl/setup.json` (0600, `ready: true`)
- Model dir: `/home/dank/models/Qwen3.8-Flash-Next-Q8_0-R9V/` — 175.30 GiB
  payload as six `Q8_0/Qwen3.8-Flash-Next-Q8_0-0000N-of-00006.gguf` shards
  + mmproj; PLE derived at the pinned size 54,400,261,120 B
- Docker→podman shim: `/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker`
  (PATH-prepended by the serve script). The R9V tooling is docker-pinned;
  the shim carries three laws:
  1. **Image ID law** — the bundle is a containerd/OCI docker-save; docker
     keys the image by its OCI MANIFEST digest (`sha256:2dac17a2…`), podman
     by config digest but preserves the manifest digest in **RepoDigests**.
     The shim answers `image inspect sha256:X` from RepoDigests and rewrites
     bare `sha256:X` run refs to `repo@sha256:X`.
  2. **LogConfig law** — podman inspect answers
     `HostConfig.LogConfig.Config: null` where docker returns the log-opts
     map (the doctor crashes on `.get("Config", {}) == null`). The shim
     records `run --log-opt` pairs and normalizes inspect output.
  3. **docker info law** — podman lacks `SecurityOptions`/`ServerVersion`;
     the shim answers rootless + `27.5.1` (liveness probe).
- venv: `~/.r9v-venv` (huggingface_hub for fetch/verify)

## Bring-up battery (qualification, 2026-09-27)

- Setup `ready: true`; all 20 payload artifacts size+sha256 verified
  (175.30 GiB total across the six shards).
- Vision: 64×64 red PNG → "Red" (79 tok, 7.4 s end-to-end on the scratch
  port). Prose: 100 tok in 9.1 s ≈ 11.0 tok/s wall **on a cold cache with
  reasoning tokens included in the wall clock** — decode-rate context, not
  a bench (no throughput bench was run, per Frank's direction).
- MTP4: mean acceptance length 3.19–5.00 across the battery.
- PLE pinned proof: `mapping_bytes = mapping_rss_bytes =
  54,400,261,120` (fully RSS-resident); doubling for re-pin would need
  108,800,522,240 ≤ available 140,876,808,192; registration took 446 s.

## Memory shape (measured)

- Load: 21.84 GiB / worker at ~158 s; cold start → serve ~16 min.
- PLE pinned residency 54.4 GB in host RAM (vs the uncensored payload's
  ssd-mmap residency).
- Context envelope: `R9V_MAX_MODEL_LEN=131072`; harness `models.yml`
  carries `contextWindow: 131072`, `maxTokens: 36864` (compaction-gap
  math: 70% × 131,072 = 91,750 + 36,864 = 128,614 ≤ 131,072). The KV
  envelope at 131,072 is smaller than the uncensored 196,608 shape.
- Prefix caching OFF at cutover (`R9V_ENABLE_PREFIX_CACHING=0`), CACHE192
  off — superseded 2026-09-28: prefix caching is ON (see bench matrix
  below).

## Prefill / decode bench matrix (2026-09-28, scratch port 7773)

Production stopped for the window (fleet bench law); one matrix plus three
follow-up runs, fork's own bench-pp + decode_speed harness, corpus-sliced
prompts with the fork's canonical decode rounds. Baseline E0 = production
shape (chunk 4096, 80-slot sync cache, MTP4):

| Config | Override | PP8K tok/s | PP131K tok/s | Decode | Verdict |
|---|---|---|---|---|---|
| E0 | — | 954.6 | 883.0 | 141.9 ms/step ≈ 23.8 t/s eff | baseline |
| E1R | chunk 8192 | **1201.4 (+25.9%)** | 844.6 (−4.4%) | 158.5 ms (−8.5% t/s) | short-prompt win, decode tax |
| E2 | chunk 16384 | — | — | — | boots; 33.3/34.2 GiB VRAM (0.93 free) → requests hang/OOM |
| E1/E3 | chunk 8192/32768 | — | — | — | PLE registration blew the 600 s engine timeout (see law below) |
| E4 | cache async=1 | — | — | — | doctor: cache-prepare backend requires synchronous single-fill LRU |
| E5/E6 | 96/112 slots | — | — | — | doctor: capacities (48, 64, 80) only; expert-budget [108, 198] rank maxima — **80 is the policy max, prod is at it** |
| E7 | MTP depth 6 | 843.1 | 819.6 | 454–489 ms/step (3.7x worse) | depth 4 stays |

Laws from the matrix:

- **4096 is the prefill-chunk ceiling twice over**: the engine caps
  scheduled tokens by the speculative config, and E2's VRAM (0.93 GiB free
  at 16384) shows bigger chunks OOM at request time. 883–955 tok/s is what
  this payload shape delivers at long context.
- **Unqualified shapes fail mid-request, not at health** — E2 and E7 both
  answered /health 200 and then died streaming.
- **PLE registration can exceed the 600 s engine timeout**
  (`VLLM_PLE_OFFLOAD_READY_TIMEOUT`) under page-cache churn — measured 740 s
  after a bench teardown; a production boot failed on it the same morning
  (unit "Deactivated successfully", no error in journal — the serve script
  exits 0 on the wait). Fixed with `R9V_PLE_READY_TIMEOUT=1800` in
  `frank-primary.env` (forwarded by launch.sh `2de0357`).

Prefix caching (E8/E9, `R9V_ENABLE_PREFIX_CACHING=1`, same port): **zero
VRAM cost** (hits reuse freed KV blocks; load VRAM identical to E0),
cold path unchanged (first request ≈ cold bench), reuse hits large — 32k
prompt TTFT 44.2 s → **3.6–4.3 s** (9x), 131k 439 s → **6.9 s** (63x).
Hit pattern: rounds 1–2 miss, round 3+ hit (identical-prompt and
conversation-growth probes both). For the harness's resend-every-turn
traffic that is a ~40–440 s TTFT win per steady-state turn; the one-turn
miss costs nothing extra. **DEPLOYED 2026-09-28** (Frank's go-ahead): the
repo env sets `R9V_ENABLE_PREFIX_CACHING=1` (commit `41e9189`), verified
on :7770 (`--enable-prefix-caching` in the container argv, health 200,
live completion answering).

## Authoring a new profile (the qualification pipeline)

Order matters at every step:

1. **Package manifest** `packages/models/<pkg>/package.json` — target shards
   + aux artifacts with sha256. Verify every target's byte count AND sha256
   after fetch (`curl -fL -C -` per shard; never trust a fetch without `-f`).
2. **Placement bootstrap** from GGUF headers: parse all shards' tensor
   tables (a KV-only shard1 is normal for split parts; each part carries its
   own tensor table). Per-expert full per layer =
   (`ffn_gate_exps` + `ffn_up_exps` + `ffn_down_exps`)/512; rank slices
   rank0 = full×416//640, rank1 = full−rank0. Hot counts =
   floor(measured_budget / per-expert total). **Anchor budgets to the
   MEASURED hot_bytes of the qualified sibling profile, never invented.**
3. **Trim + validate**: `tools/trim_experts.py --hot-counts N,M`, then
   `expert_budget.validate_manifest(m)` +
   `expert_budget.expert_memory(m, slots, {cache_rank}, runtime=<runtime.json>)`
   (runtime/catalog are keyword-only). Declared hot_bytes must match
   computed or it raises.
4. **PLE derive** (containerized `tools/prepare_ple.py` with
   `--expected-bytes/--expected-type/--expected-shape`); setup skips
   re-derivation when the file exists with matching size.
5. **profile.env**: set values BEFORE `source dual-r9700/profile.env`
   (set-first-wins; an empty string does NOT survive `:=`).

The canonical bootstrap.json wrapper schema is exactly
`{schema: "r9v.placement.v1", id, model_package, hardware, artifact_path,
artifact_sha256, scope}` with `hardware: "dual-r9700-32gb-asymmetric"`; the
wrapper `id` MUST equal the id in the profile.json placement descriptor
(setup rejects mismatches: "does not match profile placement"). Never
fabricate qualification metadata — use
`scope: "Unranked routing bootstrap only; does not certify fit or headroom"`.

GGUF parsing laws for the bootstrap: expert tensor names are
`blk.0.ffn_up_exps.weight` — split on "." gives FOUR parts, layer number is
parts[1] (a 3-part assumption crashes `int("")`); outer-KV value-type sizes
`{0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,8:str,9:arr,10:8,11:8,12:8}`; tensor byte
extents from consecutive OFFSETS, never a type-size table (mis-buckets tails
silently); the PLE Q8_0 tensor on this arch is shape (160, 320001536),
54,400,261,120 B, an entire shard.

Setup/start: `./r9v setup <alias> --state-dir <dir> --model-dir <dir> --image
sha256:<digest> --local-image --reuse-from <sibling-model-dir>
--accept-model-license` (aux artifacts reuse byte-identical files; F16 vs
Q8 mmproj will not reuse). Start requires the SAME `--state-dir`; fixed
placement via `R9V_EXPERT_MANIFEST_PATH` skips headroom/catalog/seed raises.
The host-only doctor FAILs when another instance holds the cards (it wants
3 GiB free per rank; production at ~30 GiB/card is a hard wall) — a second
instance CANNOT coexist, so qualification requires a stop+mask window for
the incumbent, then unmask+restart.

**Qualification gate (unqualified → qualified):** first start runs the
workload battery (PONG / tool_calls / vision / long-ctx), the PLE pinning
proof (`VmLck` ≈ PLE bytes in `/proc/<container-pid>/status` when
RESIDENCY_MODE=pinned), and a VRAM headroom snapshot (floor 3.3 GiB). If
headroom is violated, lower hot counts via re-trim and re-derive
MAX_EFFECTIVE. Only then record state `ready: true` and author the runbook.

Remote-execution gotchas from the same windows: `nohup X &` inside an ssh
command races ssh teardown — the script can die before writing a byte; run
long remote scripts synchronously, or setsid + disown, or via a system
unit. zsh login shells abort on unquoted `===` separators; grep globs need
quoting through ssh+zsh layers.

## Ops

- **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve
  script re-gates everything, relaunch through the shim, waits).
- **First start after boot** re-warms ~175 GiB of page cache; expect
  minutes to /health; `TimeoutStartSec=1800`.
- **Scratch/probe**: NEVER probe against the prod state dir. Copy it, set
  `R9V_HOST_PORT` to a free port and `R9V_CONTAINER_NAME` to a scratch
  name, and stop the probe before any deploy — a probe holding :7770
  produces a false-green deploy (the 2026-09-26 lesson, guarded by the
  MainPID check).
- **Rollback to uncensored**: point
  `scripts/frank-primary.env` `R9V_STATE_FILE` back at
  `/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys
  (adapter, FULL_MUTABLE_CACHE) and the template keys, redeploy — the
  uncensored state dir and payload are intact on disk.
- **Models on the wire**: served alias `frank-primary`; thinking channel
  is `reasoning` (vLLM); tool parser `qwen3_coder`.

## Known costs (accepted at cutover)

- **Residency-halved experts** — Q8_0 doubles expert bytes vs IQ4_XS, so
  the tiered cache holds half as many experts in VRAM at the same budget;
  the probe's 11 tok/s decode was residency-bound, and the production
  number depends on the re-qualified budgets in the q8 state.
- **128K envelope** — down from the uncensored 196,608 (which itself was
  down from 262,144 on llama.cpp).
- **Reference quant** — back to the unsloth Q8_0 weights (no
  abliteration); answer-quality profile differs from the uncensored
  incumbent by design.
- **Docker-pinned fork** — updates flow through image bundles + the shim,
  never a bare container run; the launcher stays the single argv owner.