frank-primary: R9V runbook + docs cutover
New ACTIVE runbook (setup/doctor PASS=34/FAIL=0, UVA transport verified in serving workers, wire-verified alias/reasoning/metrics), index + vault entries flipped: llama.cpp b11030 endpoint RETIRED 2026-09-27 to rollback target. Topology diagram updated.
This commit is contained in:
Frank
@@ -0,0 +1,126 @@
|
||||
---
|
||||
title: frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS
|
||||
service: frank-primary
|
||||
host: dankserver
|
||||
port: 7770
|
||||
status: active
|
||||
deployed: 2026-09-27
|
||||
supersedes: "[[frank-primary-llamacpp-qwen38-flash-next-q8]]"
|
||||
tags: [llm, frank-primary, r9v, vllm, radiance, gfx1201]
|
||||
---
|
||||
|
||||
# frank-primary — R9V radiance-vLLM (dankserver, ACTIVE)
|
||||
|
||||
The Dankserver LLM endpoint is the **R9V radiance-vLLM fork**
|
||||
(`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt) serving
|
||||
`Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS` — the abliterated
|
||||
Qwen3.8-Flash-Next at IQ4_XS (92.39 GiB, 3 GGUF shards) with the blockfp8
|
||||
MTP4 head and the F16 mmproj — on **both R9700s** (gfx1201, BDFs
|
||||
`0000:03:00.0` / `0000:07:00.0`) as `frank-primary` on **:7770**.
|
||||
|
||||
Replaced llama.cpp b11030 (Qwen3.8-Flash-Next Q_8) on 2026-09-27 at Frank's
|
||||
direction; downtime window ~1 h 20 min (killed 02:47, healthy 04:11 PDT).
|
||||
Rollback target: [[frank-primary-llamacpp-qwen38-flash-next-q8]] — the
|
||||
llama.cpp trio (unit/env/serve) is at commit `4992e5a`+ (the repo-side
|
||||
wiring; host unit was the relinked one) and the GGUF payload stays on disk,
|
||||
so a rollback is a redeploy.
|
||||
|
||||
## Production shape (repo-tracked)
|
||||
|
||||
The repo carries a **thin** wiring; the engine shape lives on-host:
|
||||
|
||||
| Piece | File | Law |
|
||||
|---|---|---|
|
||||
| Unit | `scripts/frank-primary.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-primary-r9v`, `Restart=on-failure` |
|
||||
| Env | `scripts/frank-primary.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary`, container `frank-primary-r9v`, GPU BDFs |
|
||||
| Serve | `scripts/serve-frank-primary.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
|
||||
| Deployer | `scripts/deploy-remote.sh` `deploy_llm_primary` | Gates on setup state ready, image digest, PLE bytes, shard files; MainPID false-green check |
|
||||
|
||||
**Why the tuned shape lives on-host:** the profile's setup run qualified a
|
||||
104-key `R9V_*` config (tiered-expert placement, per-rank offload budgets
|
||||
146.25/78.75 GiB, CED on, MTP4, parser wiring, `MAX_MODEL_LEN=196608`). It
|
||||
is written to `/home/dank/r9v-state/uncensored/setup.json` and the serve
|
||||
script layers it UNDER the repo env (repo keys win). Copying those values
|
||||
into the repo would drift from the qualification — don't.
|
||||
|
||||
**Self-healing law:** the launcher `--detach`es; the serve script therefore
|
||||
blocks on `docker wait`. Without the wait, a crashed engine is a green
|
||||
unit. `docker wait` cannot stop the container child, hence `ExecStop`.
|
||||
|
||||
## Fork checkout + shim (on-host, outside the repo)
|
||||
|
||||
- Checkout: `/home/dank/GitPackages/radiance-vllm-qwen4exp`
|
||||
- State: `/home/dank/r9v-state/uncensored/setup.json` (0600, `ready: true`)
|
||||
- Model dir: `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS/`
|
||||
(92.39 GiB payload + `r9v-data/per_layer_token_embd.iq4_nl.bin` 26.82 GiB
|
||||
PLE, sha256 `34fa36f8…`)
|
||||
- Docker→podman shim: `/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker`
|
||||
(PATH-prepended by the serve script). The R9V tooling is docker-pinned;
|
||||
the shim carries three laws:
|
||||
1. **Image ID law** — the bundle is a containerd/OCI docker-save; docker
|
||||
keys the image by its OCI MANIFEST digest (`sha256:2dac17a2…`), podman
|
||||
by config digest but preserves the manifest digest in **RepoDigests**.
|
||||
The shim answers `image inspect sha256:X` from RepoDigests and rewrites
|
||||
bare `sha256:X` run refs to `repo@sha256:X`.
|
||||
2. **LogConfig law** — podman inspect answers
|
||||
`HostConfig.LogConfig.Config: null` where docker returns the log-opts
|
||||
map (the doctor crashes on `.get("Config", {}) == null`). The shim
|
||||
records `run --log-opt` pairs and normalizes inspect output.
|
||||
3. **docker info law** — podman lacks `SecurityOptions`/`ServerVersion`;
|
||||
the shim answers rootless + `27.5.1` (liveness probe).
|
||||
- venv: `~/.r9v-venv` (huggingface_hub for fetch/verify)
|
||||
|
||||
## Verified at cutover (2026-09-27)
|
||||
|
||||
- Setup: **PASS=28 WARN=3 FAIL=0** — 20/20 artifacts size+sha256 verified
|
||||
(~93 GiB), PLE derived at exact pinned size, CED projector verified.
|
||||
- First-start doctor battery: **PASS=34 WARN=8 FAIL=0**, including
|
||||
`runtime-worker-transport`: **pinned UVA reads, H2D copies, TP
|
||||
all-reduce verified inside serving workers** on both BDFs.
|
||||
- Wire: `/v1/models` → `frank-primary`; `/health` + `/metrics` 200
|
||||
(Prometheus scraping); "Say PONG" (non-thinking) → `'PONG'`, stop;
|
||||
thinking requests populate `reasoning` (vLLM channel).
|
||||
- Memlock: the full tiered-expert load ran to healthy **under a plain dank
|
||||
session** before the unit existed — the pin path does NOT hard-require
|
||||
`LimitMEMLOCK=infinity` on this host, but the root unit keeps fleet
|
||||
parity and headroom for the pinned reserve.
|
||||
- e2e gate: 32 passed / 1 failed — the failure is `11-admin open links`,
|
||||
which needs FrankBook-Pro (192.168.0.112) awake; the machine was down and
|
||||
WOL from both hosts failed. Suite minus that spec: **31 passed** (the
|
||||
one-time 12-crawl flake re-passes in isolation).
|
||||
|
||||
## Memory shape (measured)
|
||||
|
||||
- Both cards ~29–30 GiB used of 31.86; engine budget
|
||||
`CPU_OFFLOAD_GB_BY_DEVICE=146.25,78.75` (UVA host pin ≈ 43 GiB on rank 0
|
||||
+ 16 GiB PLE pinned reserve; PLE residency mode `ssd` mmap+trim).
|
||||
- `MemAvailable` floor: 56.3 GiB (`R9V_MIN_HOST_AVAILABLE_BYTES`).
|
||||
- Context envelope: 192,608 native window served as `MAX_MODEL_LEN=196608`;
|
||||
harness `models.yml` carries `contextWindow: 196608`, `maxTokens: 57344`
|
||||
(compaction-gap math: 70% × 196,608 = 137,625 + 57,344 ≤ 196,608).
|
||||
|
||||
## Ops
|
||||
|
||||
- **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve
|
||||
script re-gates everything, relaunch through the shim, waits).
|
||||
- **First start after boot** re-warms ~93 GiB of page cache; expect several
|
||||
minutes to /health; `TimeoutStartSec=1800`.
|
||||
- **Scratch/probe**: NEVER probe against the prod state dir. Copy it, set
|
||||
`R9V_HOST_PORT` to a free port and `R9V_CONTAINER_NAME` to a scratch
|
||||
name, and stop the probe before any deploy — a probe holding :7770
|
||||
produces a false-green deploy (the 2026-09-26 lesson, now guarded by the
|
||||
MainPID check).
|
||||
- **Qualification artifacts** (kept on-host):
|
||||
`/home/dank/r9v-state/uncensored/qualification-*`,
|
||||
failure bundles `failure-*` (review before sharing — raw logs).
|
||||
- **Models on the wire**: served alias `frank-primary`; thinking channel is
|
||||
`reasoning` (vLLM), tool parser `qwen3_coder` (not qwen3_xml like the
|
||||
vllm030 wiring had planned).
|
||||
|
||||
## Known costs (accepted at selection)
|
||||
|
||||
- **Abliterated weights** (Dyluhn uncensored build) — not the reference
|
||||
model; answer-quality deltas are possible.
|
||||
- **192K envelope** — down from 262,144 on llama.cpp.
|
||||
- **Docker-pinned fork** — updates flow through image bundles + the shim,
|
||||
never a bare container run; the launcher stays the single argv owner.
|
||||
@@ -1,14 +1,14 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-09-26):
|
||||
**llama.cpp on all three endpoints** — Dankserver runs unsloth llama.cpp
|
||||
(the b11030-mix prebuilt, ROCm gfx120X) serving frank-primary
|
||||
(unsloth Qwen3.8-Flash-Next Q8_0 + the shared Q8_0 MTP head + ngram-mod +
|
||||
vision) across both Radeon AI PRO R9700s on :7770, and Dankclaw runs two
|
||||
endpoints — stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large,
|
||||
unsloth Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the
|
||||
gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS
|
||||
Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771).
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-09-27):
|
||||
**Dankserver runs R9V radiance-vLLM on frank-primary** (the drwolfen fork
|
||||
serving the abliterated Dyluhn Qwen3.8-Flash-Next-Uncensored IQ4_XS + MTP4
|
||||
across both Radeon AI PRO R9700s on :7770, docker-pinned runtime driven
|
||||
through a docker→podman shim), and Dankclaw runs two llama.cpp endpoints —
|
||||
stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth
|
||||
Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151
|
||||
iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B
|
||||
QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771).
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -22,10 +22,11 @@ graph LR
|
||||
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
|
||||
A[2x Radeon AI PRO R9700
|
||||
gfx1201, 32 GiB each]
|
||||
B2[unsloth llama.cpp b11030-mix, ROCm gfx120X
|
||||
B2[R9V radiance-vLLM fork, docker-pinned
|
||||
via docker-podman shim
|
||||
frank-primary, port 7770]
|
||||
C[Qwen3.8-Flash-Next Q8_0 + shared-Q8_0 MTP head
|
||||
layer split on both cards, 262144 ctx / 1 slot, ngram-mod, vision]
|
||||
C[Qwen3.8-Flash-Next-Uncensored IQ4_XS
|
||||
TP=2 tiered experts, MTP4, CED, 196608 ctx, vision]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
@@ -57,8 +58,19 @@ frank-small, port 7771]
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — the ACTIVE
|
||||
frank-primary runbook (2026-09-26, root systemd unit, port 7770):
|
||||
- [[frank-primary-r9v-radiance-qwen38-uncensored]] — the ACTIVE
|
||||
frank-primary runbook (2026-09-27, root systemd unit, port 7770): the
|
||||
R9V radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned
|
||||
image bundle `2dac17a2`, driven through a docker→podman shim) serving
|
||||
Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
|
||||
blockfp8 MTP4 (depth 4) + F16 vision on BOTH R9700s — TP=2 tiered
|
||||
experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428
|
||||
static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx, reasoning
|
||||
channel `reasoning`, parser `qwen3_coder`. Root unit + LimitMEMLOCK;
|
||||
the tuned 104-key shape lives in the on-host setup state. Rollback:
|
||||
[[frank-primary-llamacpp-qwen38-flash-next-q8]]
|
||||
- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
|
||||
(rollback target; the llama.cpp b11030 endpoint of 2026-09-26):
|
||||
unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime)
|
||||
serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s —
|
||||
routed experts and the 51B-param n-gram table computed in host RAM
|
||||
|
||||
@@ -124,8 +124,15 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
|
||||
- [[deployment]] — unified deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — the ACTIVE
|
||||
frank-primary runbook: unsloth llama.cpp (b11030-mix prebuilt, ROCm
|
||||
- [[frank-primary-r9v-radiance-qwen38-uncensored]] — the ACTIVE
|
||||
frank-primary runbook: the R9V radiance-vLLM fork (docker-pinned bundle
|
||||
`2dac17a2`, docker→podman shim) serving the abliterated
|
||||
Dyluhn Qwen3.8-Flash-Next-Uncensored IQ4_XS (92.39 GiB) + MTP4 + F16
|
||||
vision on Dankserver's both R9700s — TP=2 tiered experts with a UVA
|
||||
pinned host copy, PLE on ssd-mmap, CED on, 196608 ctx, root unit with
|
||||
LimitMEMLOCK, the on-host 104-key tuned state, cutover and rollback
|
||||
- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
|
||||
(rollback target): unsloth llama.cpp (b11030-mix prebuilt, ROCm
|
||||
gfx120X, root unit) serving unsloth Qwen3.8-Flash-Next at Q8_0 (175.3
|
||||
GiB) on Dankserver's both R9700s — the shared Q8_0 MTP head chained
|
||||
behind ngram-mod, BF16 vision, the experts and the 51B n-gram table
|
||||
|
||||
Reference in New Issue
Block a user