serverAppConfigs/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md

title, service, host, port, status, deployed, supersedes, tags

title	service	host	port	status	deployed	supersedes	tags
frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0	frank-primary	dankserver	7770	active	2026-09-27	frank-primary-r9v-radiance-qwen38-uncensored	llm frank-primary r9v vllm radiance gfx1201 q8

frank-primary — R9V radiance-vLLM Q8_0 (dankserver, ACTIVE)

The Dankserver LLM endpoint is the R9V radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned prebuilt image bundle 2dac17a2) serving the unsloth Qwen3.8-Flash-Next-GGUF Q8_0 payload (175.30 GiB, 6 GGUF shards, sha256-verified per part) with the blockfp8 MTP4 head and the mmproj, on both R9700s (gfx1201, BDFs 0000:03:00.0 / 0000:07:00.0) as frank-primary on :7770.

Cut over from the uncensored IQ4_XS payload on 2026-09-27 at Frank's direction. Rollback target: frank-primary-r9v-radiance-qwen38-uncensored — the uncensored state dir (/home/dank/r9v-state/uncensored/, ready: true), its frozen config (restore-config-20260927.env) and payload stay on disk; a rollback is an env-file R9V_STATE_FILE flip plus a redeploy.

Why the swap

The uncensored IQ4_XS payload measured 59–96 tok/s decode with draft-acceptance swings; the Q8_0 payload trades VRAM-resident expert headroom for precision — quantization to Q8_0 doubles expert bytes per block, so the tiered-expert residency budget halves — the probe-era 11 tok/s decode was residency-bound (expert streaming, not compute). The "Known costs" section below carries the accepted trade.

Production shape (repo-tracked)

The repo carries a thin wiring; the engine shape lives on-host:

Piece	File	Law
Unit	scripts/frank-primary.service	Root unit, User=dank, LimitMEMLOCK=infinity, ExecStop=podman stop -t 120 frank-primary-r9v, Restart=on-failure
Env	scripts/frank-primary.env	Production overrides ONLY: bind 0.0.0.0, :7770, alias frank-primary, container frank-primary-r9v, GPU BDFs, dev-overlay adapter, R9V_FULL_MUTABLE_CACHE=0, R9V_PLE_READY_TIMEOUT=1800, prefix caching ON, sharp chat template
Serve	scripts/serve-frank-primary.sh	Layers setup state under repo env → execs the fork's own scripts/launch.sh via the docker→podman shim → docker wait
Deployer	scripts/deploy-remote.sh deploy_llm_primary	Gates on setup state ready, image digest, PLE bytes, all six shard files + mmproj; MainPID false-green check

Why the tuned shape lives on-host: the q8 profile's setup run qualified the R9V_* config (tiered-expert placement, per-rank offload budgets, 80-slot tiered cache, PLE pinned residency 54.4 GB, MTP4, parser wiring, MAX_MODEL_LEN=131072). It is written to /home/dank/r9v-state/q8xl/setup.json and the serve script layers it UNDER the repo env (repo keys win). Copying those values into the repo would drift from the qualification — don't.

Spec decode: MTP4 only — the ngram-mod ask is NOT expressible

llama.cpp chains drafter types (--spec-type draft-mtp,ngram-mod on frank-small/large). The fork's speculative_config is a single dict (one method per engine): mtp, ngram, ngram_gpu, dflash, … are mutually exclusive — no chaining seam exists in the fork. What IS active:

• MTP4 (R9V_MTP_SPEC_TOKENS=4, blockfp8 draft head, draft TP=2) — measured mean acceptance length 3.19–5.00 during qualification.
• The PLE n-gram — architectural (ngram_embedding module), rides along automatically; it is not the llama.cpp ngram-mod matcher and cannot be "added" as a second spec method.
• ngram_gpu exists as an ALTERNATIVE single method (fork's GPU n-gram proposer) — swapping MTP4 for it is an either/or, not a chain.

Sharp chat template (fleet parity with frank-large)

scripts/qwen-sharp-v22.4.0.jinja is served via the launcher's opt-in R9V_CHAT_TEMPLATE_FILE passthrough (fork commit 30d4a32): the file is mounted read-only at /opt/r9v/share/chat-template.jinja and passed as --chat-template, which wins over the tokenizer dir's metadata/chat_template.jinja; R9V_CHAT_TEMPLATE_KWARGS carries --default-chat-template-kwargs {"preserve_thinking":true} (the template defaults preserve_thinking true natively; request-level kwargs still win per request). The template folds reasoning efforts (high/xhigh/max → xhigh, minimal/low → low), matching models.yml's reasoningEffortMap. The sha-pinned metadata/chat_template.jinja in the package is NEVER edited in place — mutating it drifts the qualification.

Two q8-only runtime laws

1. Quantized inject shard — the Q8_0 payload quantizes BOTH fused hyper-connection shards (the IQ4_XS package ships inject as F32), so the shard-uniformity check rejects the fused projection at load. R9V_DEV_QWEN4_EXP_ADAPTER_PY (fork commit 363d606) mounts a dev-overlay adapter that dequantizes a quantized inject shard in-stream; uncensored payloads pass through unchanged.
2. R9V_FULL_MUTABLE_CACHE=0 — the runtime overlay's env.always injects the uncensored 160-slot full-mutable cache geometry, which the q8 profile's 80-slot tiered cache fails with a baseline mismatch. The launcher passes this key AFTER the overlay envs (docker last-wins).

Fork checkout + shim (on-host, outside the repo)

• Checkout: /home/dank/GitPackages/radiance-vllm-qwen4exp (fork commits 363d606 dev-overlay, 2fe6dbb, 50a79ec qualification doc, 30d4a32 chat-template passthrough, 2de0357 PLE-timeout forwarding — local-only, remote is GitHub)
• State: /home/dank/r9v-state/q8xl/setup.json (0600, ready: true)
• Model dir: /home/dank/models/Qwen3.8-Flash-Next-Q8_0-R9V/ — 175.30 GiB payload as six Q8_0/Qwen3.8-Flash-Next-Q8_0-0000N-of-00006.gguf shards
  • mmproj; PLE derived at the pinned size 54,400,261,120 B
• Docker→podman shim: /home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker (PATH-prepended by the serve script). The R9V tooling is docker-pinned; the shim carries three laws:
  1. Image ID law — the bundle is a containerd/OCI docker-save; docker keys the image by its OCI MANIFEST digest (sha256:2dac17a2…), podman by config digest but preserves the manifest digest in RepoDigests. The shim answers image inspect sha256:X from RepoDigests and rewrites bare sha256:X run refs to repo@sha256:X.
  2. LogConfig law — podman inspect answers HostConfig.LogConfig.Config: null where docker returns the log-opts map (the doctor crashes on .get("Config", {}) == null). The shim records run --log-opt pairs and normalizes inspect output.
  3. docker info law — podman lacks SecurityOptions/ServerVersion; the shim answers rootless + 27.5.1 (liveness probe).
• venv: ~/.r9v-venv (huggingface_hub for fetch/verify)

Bring-up battery (qualification, 2026-09-27)

• Setup ready: true; all 20 payload artifacts size+sha256 verified (175.30 GiB total across the six shards).
• Vision: 64×64 red PNG → "Red" (79 tok, 7.4 s end-to-end on the scratch port). Prose: 100 tok in 9.1 s ≈ 11.0 tok/s wall on a cold cache with reasoning tokens included in the wall clock — decode-rate context, not a bench (no throughput bench was run, per Frank's direction).
• MTP4: mean acceptance length 3.19–5.00 across the battery.
• PLE pinned proof: mapping_bytes = mapping_rss_bytes = 54,400,261,120 (fully RSS-resident); doubling for re-pin would need 108,800,522,240 ≤ available 140,876,808,192; registration took 446 s.

Memory shape (measured)

• Load: 21.84 GiB / worker at ~158 s; cold start → serve ~16 min.
• PLE pinned residency 54.4 GB in host RAM (vs the uncensored payload's ssd-mmap residency).
• Context envelope: R9V_MAX_MODEL_LEN=131072; harness models.yml carries contextWindow: 131072, maxTokens: 36864 (compaction-gap math: 70% × 131,072 = 91,750 + 36,864 = 128,614 ≤ 131,072). The KV envelope at 131,072 is smaller than the uncensored 196,608 shape.
• Prefix caching OFF at cutover (R9V_ENABLE_PREFIX_CACHING=0), CACHE192 off — superseded 2026-09-28: prefix caching is ON (see bench matrix below).

Prefill / decode bench matrix (2026-09-28, scratch port 7773)

Production stopped for the window (fleet bench law); one matrix plus three follow-up runs, fork's own bench-pp + decode_speed harness, corpus-sliced prompts with the fork's canonical decode rounds. Baseline E0 = production shape (chunk 4096, 80-slot sync cache, MTP4):

Config	Override	PP8K tok/s	PP131K tok/s	Decode	Verdict
E0	—	954.6	883.0	141.9 ms/step ≈ 23.8 t/s eff	baseline
E1R	chunk 8192	1201.4 (+25.9%)	844.6 (−4.4%)	158.5 ms (−8.5% t/s)	short-prompt win, decode tax
E2	chunk 16384	—	—	—	boots; 33.3/34.2 GiB VRAM (0.93 free) → requests hang/OOM
E1/E3	chunk 8192/32768	—	—	—	PLE registration blew the 600 s engine timeout (see law below)
E4	cache async=1	—	—	—	doctor: cache-prepare backend requires synchronous single-fill LRU
E5/E6	96/112 slots	—	—	—	doctor: capacities (48, 64, 80) only; expert-budget [108, 198] rank maxima — 80 is the policy max, prod is at it
E7	MTP depth 6	843.1	819.6	454–489 ms/step (3.7x worse)	depth 4 stays

Laws from the matrix:

• 4096 is the prefill-chunk ceiling twice over: the engine caps scheduled tokens by the speculative config, and E2's VRAM (0.93 GiB free at 16384) shows bigger chunks OOM at request time. 883–955 tok/s is what this payload shape delivers at long context.
• Unqualified shapes fail mid-request, not at health — E2 and E7 both answered /health 200 and then died streaming.
• PLE registration can exceed the 600 s engine timeout (VLLM_PLE_OFFLOAD_READY_TIMEOUT) under page-cache churn — measured 740 s after a bench teardown; a production boot failed on it the same morning (unit "Deactivated successfully", no error in journal — the serve script exits 0 on the wait). Fixed with R9V_PLE_READY_TIMEOUT=1800 in frank-primary.env (forwarded by launch.sh 2de0357).

Prefix caching (E8/E9, R9V_ENABLE_PREFIX_CACHING=1, same port): zero VRAM cost (hits reuse freed KV blocks; load VRAM identical to E0), cold path unchanged (first request ≈ cold bench), reuse hits large — 32k prompt TTFT 44.2 s → 3.6–4.3 s (9x), 131k 439 s → 6.9 s (63x). Hit pattern: rounds 1–2 miss, round 3+ hit (identical-prompt and conversation-growth probes both). For the harness's resend-every-turn traffic that is a ~40–440 s TTFT win per steady-state turn; the one-turn miss costs nothing extra. DEPLOYED 2026-09-28 (Frank's go-ahead): the repo env sets R9V_ENABLE_PREFIX_CACHING=1 (commit 41e9189), verified on :7770 (--enable-prefix-caching in the container argv, health 200, live completion answering).

Authoring a new profile (the qualification pipeline)

Order matters at every step:

1. Package manifest packages/models/<pkg>/package.json — target shards
   • aux artifacts with sha256. Verify every target's byte count AND sha256 after fetch (curl -fL -C - per shard; never trust a fetch without -f).
2. Placement bootstrap from GGUF headers: parse all shards' tensor tables (a KV-only shard1 is normal for split parts; each part carries its own tensor table). Per-expert full per layer = (ffn_gate_exps + ffn_up_exps + ffn_down_exps)/512; rank slices rank0 = full×416//640, rank1 = full−rank0. Hot counts = floor(measured_budget / per-expert total). Anchor budgets to the MEASURED hot_bytes of the qualified sibling profile, never invented.
3. Trim + validate: tools/trim_experts.py --hot-counts N,M, then expert_budget.validate_manifest(m) + expert_budget.expert_memory(m, slots, {cache_rank}, runtime=<runtime.json>) (runtime/catalog are keyword-only). Declared hot_bytes must match computed or it raises.
4. PLE derive (containerized tools/prepare_ple.py with --expected-bytes/--expected-type/--expected-shape); setup skips re-derivation when the file exists with matching size.
5. profile.env: set values BEFORE source dual-r9700/profile.env (set-first-wins; an empty string does NOT survive :=).

The canonical bootstrap.json wrapper schema is exactly {schema: "r9v.placement.v1", id, model_package, hardware, artifact_path, artifact_sha256, scope} with hardware: "dual-r9700-32gb-asymmetric"; the wrapper id MUST equal the id in the profile.json placement descriptor (setup rejects mismatches: "does not match profile placement"). Never fabricate qualification metadata — use scope: "Unranked routing bootstrap only; does not certify fit or headroom".

GGUF parsing laws for the bootstrap: expert tensor names are blk.0.ffn_up_exps.weight — split on "." gives FOUR parts, layer number is parts[1] (a 3-part assumption crashes int("")); outer-KV value-type sizes {0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,8:str,9:arr,10:8,11:8,12:8}; tensor byte extents from consecutive OFFSETS, never a type-size table (mis-buckets tails silently); the PLE Q8_0 tensor on this arch is shape (160, 320001536), 54,400,261,120 B, an entire shard.

Setup/start: ./r9v setup <alias> --state-dir <dir> --model-dir <dir> --image sha256:<digest> --local-image --reuse-from <sibling-model-dir> --accept-model-license (aux artifacts reuse byte-identical files; F16 vs Q8 mmproj will not reuse). Start requires the SAME --state-dir; fixed placement via R9V_EXPERT_MANIFEST_PATH skips headroom/catalog/seed raises. The host-only doctor FAILs when another instance holds the cards (it wants 3 GiB free per rank; production at ~30 GiB/card is a hard wall) — a second instance CANNOT coexist, so qualification requires a stop+mask window for the incumbent, then unmask+restart.

Qualification gate (unqualified → qualified): first start runs the workload battery (PONG / tool_calls / vision / long-ctx), the PLE pinning proof (VmLck ≈ PLE bytes in /proc/<container-pid>/status when RESIDENCY_MODE=pinned), and a VRAM headroom snapshot (floor 3.3 GiB). If headroom is violated, lower hot counts via re-trim and re-derive MAX_EFFECTIVE. Only then record state ready: true and author the runbook.

Remote-execution gotchas from the same windows: nohup X & inside an ssh command races ssh teardown — the script can die before writing a byte; run long remote scripts synchronously, or setsid + disown, or via a system unit. zsh login shells abort on unquoted === separators; grep globs need quoting through ssh+zsh layers.

Ops

• Restart: sudo systemctl restart frank-primary (root unit; the serve script re-gates everything, relaunch through the shim, waits).
• First start after boot re-warms ~175 GiB of page cache; expect minutes to /health; TimeoutStartSec=1800.
• Scratch/probe: NEVER probe against the prod state dir. Copy it, set R9V_HOST_PORT to a free port and R9V_CONTAINER_NAME to a scratch name, and stop the probe before any deploy — a probe holding :7770 produces a false-green deploy (the 2026-09-26 lesson, guarded by the MainPID check).
• Rollback to uncensored: point scripts/frank-primary.env R9V_STATE_FILE back at /home/dank/r9v-state/uncensored/setup.json, drop the two q8-only keys (adapter, FULL_MUTABLE_CACHE) and the template keys, redeploy — the uncensored state dir and payload are intact on disk.
• Models on the wire: served alias frank-primary; thinking channel is reasoning (vLLM); tool parser qwen3_coder.

Known costs (accepted at cutover)

• Residency-halved experts — Q8_0 doubles expert bytes vs IQ4_XS, so the tiered cache holds half as many experts in VRAM at the same budget; the probe's 11 tok/s decode was residency-bound, and the production number depends on the re-qualified budgets in the q8 state.
• 128K envelope — down from the uncensored 196,608 (which itself was down from 262,144 on llama.cpp).
• Reference quant — back to the unsloth Q8_0 weights (no abliteration); answer-quality profile differs from the uncensored incumbent by design.
• Docker-pinned fork — updates flow through image bundles + the shim, never a bare container run; the launcher stays the single argv owner.