Frank
ca6ba6d0e9
New ACTIVE runbook (setup/doctor PASS=34/FAIL=0, UVA transport verified in serving workers, wire-verified alias/reasoning/metrics), index + vault entries flipped: llama.cpp b11030 endpoint RETIRED 2026-09-27 to rollback target. Topology diagram updated.
127 lines
6.7 KiB
Markdown
127 lines
6.7 KiB
Markdown
---
|
||
title: frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS
|
||
service: frank-primary
|
||
host: dankserver
|
||
port: 7770
|
||
status: active
|
||
deployed: 2026-09-27
|
||
supersedes: "[[frank-primary-llamacpp-qwen38-flash-next-q8]]"
|
||
tags: [llm, frank-primary, r9v, vllm, radiance, gfx1201]
|
||
---
|
||
|
||
# frank-primary — R9V radiance-vLLM (dankserver, ACTIVE)
|
||
|
||
The Dankserver LLM endpoint is the **R9V radiance-vLLM fork**
|
||
(`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt) serving
|
||
`Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS` — the abliterated
|
||
Qwen3.8-Flash-Next at IQ4_XS (92.39 GiB, 3 GGUF shards) with the blockfp8
|
||
MTP4 head and the F16 mmproj — on **both R9700s** (gfx1201, BDFs
|
||
`0000:03:00.0` / `0000:07:00.0`) as `frank-primary` on **:7770**.
|
||
|
||
Replaced llama.cpp b11030 (Qwen3.8-Flash-Next Q_8) on 2026-09-27 at Frank's
|
||
direction; downtime window ~1 h 20 min (killed 02:47, healthy 04:11 PDT).
|
||
Rollback target: [[frank-primary-llamacpp-qwen38-flash-next-q8]] — the
|
||
llama.cpp trio (unit/env/serve) is at commit `4992e5a`+ (the repo-side
|
||
wiring; host unit was the relinked one) and the GGUF payload stays on disk,
|
||
so a rollback is a redeploy.
|
||
|
||
## Production shape (repo-tracked)
|
||
|
||
The repo carries a **thin** wiring; the engine shape lives on-host:
|
||
|
||
| Piece | File | Law |
|
||
|---|---|---|
|
||
| Unit | `scripts/frank-primary.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-primary-r9v`, `Restart=on-failure` |
|
||
| Env | `scripts/frank-primary.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary`, container `frank-primary-r9v`, GPU BDFs |
|
||
| Serve | `scripts/serve-frank-primary.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
|
||
| Deployer | `scripts/deploy-remote.sh` `deploy_llm_primary` | Gates on setup state ready, image digest, PLE bytes, shard files; MainPID false-green check |
|
||
|
||
**Why the tuned shape lives on-host:** the profile's setup run qualified a
|
||
104-key `R9V_*` config (tiered-expert placement, per-rank offload budgets
|
||
146.25/78.75 GiB, CED on, MTP4, parser wiring, `MAX_MODEL_LEN=196608`). It
|
||
is written to `/home/dank/r9v-state/uncensored/setup.json` and the serve
|
||
script layers it UNDER the repo env (repo keys win). Copying those values
|
||
into the repo would drift from the qualification — don't.
|
||
|
||
**Self-healing law:** the launcher `--detach`es; the serve script therefore
|
||
blocks on `docker wait`. Without the wait, a crashed engine is a green
|
||
unit. `docker wait` cannot stop the container child, hence `ExecStop`.
|
||
|
||
## Fork checkout + shim (on-host, outside the repo)
|
||
|
||
- Checkout: `/home/dank/GitPackages/radiance-vllm-qwen4exp`
|
||
- State: `/home/dank/r9v-state/uncensored/setup.json` (0600, `ready: true`)
|
||
- Model dir: `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS/`
|
||
(92.39 GiB payload + `r9v-data/per_layer_token_embd.iq4_nl.bin` 26.82 GiB
|
||
PLE, sha256 `34fa36f8…`)
|
||
- Docker→podman shim: `/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker`
|
||
(PATH-prepended by the serve script). The R9V tooling is docker-pinned;
|
||
the shim carries three laws:
|
||
1. **Image ID law** — the bundle is a containerd/OCI docker-save; docker
|
||
keys the image by its OCI MANIFEST digest (`sha256:2dac17a2…`), podman
|
||
by config digest but preserves the manifest digest in **RepoDigests**.
|
||
The shim answers `image inspect sha256:X` from RepoDigests and rewrites
|
||
bare `sha256:X` run refs to `repo@sha256:X`.
|
||
2. **LogConfig law** — podman inspect answers
|
||
`HostConfig.LogConfig.Config: null` where docker returns the log-opts
|
||
map (the doctor crashes on `.get("Config", {}) == null`). The shim
|
||
records `run --log-opt` pairs and normalizes inspect output.
|
||
3. **docker info law** — podman lacks `SecurityOptions`/`ServerVersion`;
|
||
the shim answers rootless + `27.5.1` (liveness probe).
|
||
- venv: `~/.r9v-venv` (huggingface_hub for fetch/verify)
|
||
|
||
## Verified at cutover (2026-09-27)
|
||
|
||
- Setup: **PASS=28 WARN=3 FAIL=0** — 20/20 artifacts size+sha256 verified
|
||
(~93 GiB), PLE derived at exact pinned size, CED projector verified.
|
||
- First-start doctor battery: **PASS=34 WARN=8 FAIL=0**, including
|
||
`runtime-worker-transport`: **pinned UVA reads, H2D copies, TP
|
||
all-reduce verified inside serving workers** on both BDFs.
|
||
- Wire: `/v1/models` → `frank-primary`; `/health` + `/metrics` 200
|
||
(Prometheus scraping); "Say PONG" (non-thinking) → `'PONG'`, stop;
|
||
thinking requests populate `reasoning` (vLLM channel).
|
||
- Memlock: the full tiered-expert load ran to healthy **under a plain dank
|
||
session** before the unit existed — the pin path does NOT hard-require
|
||
`LimitMEMLOCK=infinity` on this host, but the root unit keeps fleet
|
||
parity and headroom for the pinned reserve.
|
||
- e2e gate: 32 passed / 1 failed — the failure is `11-admin open links`,
|
||
which needs FrankBook-Pro (192.168.0.112) awake; the machine was down and
|
||
WOL from both hosts failed. Suite minus that spec: **31 passed** (the
|
||
one-time 12-crawl flake re-passes in isolation).
|
||
|
||
## Memory shape (measured)
|
||
|
||
- Both cards ~29–30 GiB used of 31.86; engine budget
|
||
`CPU_OFFLOAD_GB_BY_DEVICE=146.25,78.75` (UVA host pin ≈ 43 GiB on rank 0
|
||
+ 16 GiB PLE pinned reserve; PLE residency mode `ssd` mmap+trim).
|
||
- `MemAvailable` floor: 56.3 GiB (`R9V_MIN_HOST_AVAILABLE_BYTES`).
|
||
- Context envelope: 192,608 native window served as `MAX_MODEL_LEN=196608`;
|
||
harness `models.yml` carries `contextWindow: 196608`, `maxTokens: 57344`
|
||
(compaction-gap math: 70% × 196,608 = 137,625 + 57,344 ≤ 196,608).
|
||
|
||
## Ops
|
||
|
||
- **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve
|
||
script re-gates everything, relaunch through the shim, waits).
|
||
- **First start after boot** re-warms ~93 GiB of page cache; expect several
|
||
minutes to /health; `TimeoutStartSec=1800`.
|
||
- **Scratch/probe**: NEVER probe against the prod state dir. Copy it, set
|
||
`R9V_HOST_PORT` to a free port and `R9V_CONTAINER_NAME` to a scratch
|
||
name, and stop the probe before any deploy — a probe holding :7770
|
||
produces a false-green deploy (the 2026-09-26 lesson, now guarded by the
|
||
MainPID check).
|
||
- **Qualification artifacts** (kept on-host):
|
||
`/home/dank/r9v-state/uncensored/qualification-*`,
|
||
failure bundles `failure-*` (review before sharing — raw logs).
|
||
- **Models on the wire**: served alias `frank-primary`; thinking channel is
|
||
`reasoning` (vLLM), tool parser `qwen3_coder` (not qwen3_xml like the
|
||
vllm030 wiring had planned).
|
||
|
||
## Known costs (accepted at selection)
|
||
|
||
- **Abliterated weights** (Dyluhn uncensored build) — not the reference
|
||
model; answer-quality deltas are possible.
|
||
- **192K envelope** — down from 262,144 on llama.cpp.
|
||
- **Docker-pinned fork** — updates flow through image bundles + the shim,
|
||
never a bare container run; the launcher stays the single argv owner.
|