frank-primary: R9V runbook + docs cutover

New ACTIVE runbook (setup/doctor PASS=34/FAIL=0, UVA transport verified in serving workers, wire-verified alias/reasoning/metrics), index + vault entries flipped: llama.cpp b11030 endpoint RETIRED 2026-09-27 to rollback target. Topology diagram updated.
2026-09-27 04:35:05 -07:00
parent 7562e41ba4
commit ca6ba6d0e9
3 changed files with 161 additions and 16 deletions
@@ -0,0 +1,126 @@
+---
+title: frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS
+service: frank-primary
+host: dankserver
+port: 7770
+status: active
+deployed: 2026-09-27
+supersedes: "[[frank-primary-llamacpp-qwen38-flash-next-q8]]"
+tags: [llm, frank-primary, r9v, vllm, radiance, gfx1201]
+---
+
+# frank-primary — R9V radiance-vLLM (dankserver, ACTIVE)
+
+The Dankserver LLM endpoint is the **R9V radiance-vLLM fork**
+(`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt) serving
+`Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS` — the abliterated
+Qwen3.8-Flash-Next at IQ4_XS (92.39 GiB, 3 GGUF shards) with the blockfp8
+MTP4 head and the F16 mmproj — on **both R9700s** (gfx1201, BDFs
+`0000:03:00.0` / `0000:07:00.0`) as `frank-primary` on **:7770**.
+
+Replaced llama.cpp b11030 (Qwen3.8-Flash-Next Q_8) on 2026-09-27 at Frank's
+direction; downtime window ~1 h 20 min (killed 02:47, healthy 04:11 PDT).
+Rollback target: [[frank-primary-llamacpp-qwen38-flash-next-q8]] — the
+llama.cpp trio (unit/env/serve) is at commit `4992e5a`+ (the repo-side
+wiring; host unit was the relinked one) and the GGUF payload stays on disk,
+so a rollback is a redeploy.
+
+## Production shape (repo-tracked)
+
+The repo carries a **thin** wiring; the engine shape lives on-host:
+
+| Piece | File | Law |
+|---|---|---|
+| Unit | `scripts/frank-primary.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-primary-r9v`, `Restart=on-failure` |
+| Env | `scripts/frank-primary.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary`, container `frank-primary-r9v`, GPU BDFs |
+| Serve | `scripts/serve-frank-primary.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
+| Deployer | `scripts/deploy-remote.sh` `deploy_llm_primary` | Gates on setup state ready, image digest, PLE bytes, shard files; MainPID false-green check |
+
+**Why the tuned shape lives on-host:** the profile's setup run qualified a
+104-key `R9V_*` config (tiered-expert placement, per-rank offload budgets
+146.25/78.75 GiB, CED on, MTP4, parser wiring, `MAX_MODEL_LEN=196608`). It
+is written to `/home/dank/r9v-state/uncensored/setup.json` and the serve
+script layers it UNDER the repo env (repo keys win). Copying those values
+into the repo would drift from the qualification — don't.
+
+**Self-healing law:** the launcher `--detach`es; the serve script therefore
+blocks on `docker wait`. Without the wait, a crashed engine is a green
+unit. `docker wait` cannot stop the container child, hence `ExecStop`.
+
+## Fork checkout + shim (on-host, outside the repo)
+
+- Checkout: `/home/dank/GitPackages/radiance-vllm-qwen4exp`
+- State: `/home/dank/r9v-state/uncensored/setup.json` (0600, `ready: true`)
+- Model dir: `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS/`
+  (92.39 GiB payload + `r9v-data/per_layer_token_embd.iq4_nl.bin` 26.82 GiB
+  PLE, sha256 `34fa36f8…`)
+- Docker→podman shim: `/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker`
+  (PATH-prepended by the serve script). The R9V tooling is docker-pinned;
+  the shim carries three laws:
+  1. **Image ID law** — the bundle is a containerd/OCI docker-save; docker
+     keys the image by its OCI MANIFEST digest (`sha256:2dac17a2…`), podman
+     by config digest but preserves the manifest digest in **RepoDigests**.
+     The shim answers `image inspect sha256:X` from RepoDigests and rewrites
+     bare `sha256:X` run refs to `repo@sha256:X`.
+  2. **LogConfig law** — podman inspect answers
+     `HostConfig.LogConfig.Config: null` where docker returns the log-opts
+     map (the doctor crashes on `.get("Config", {}) == null`). The shim
+     records `run --log-opt` pairs and normalizes inspect output.
+  3. **docker info law** — podman lacks `SecurityOptions`/`ServerVersion`;
+     the shim answers rootless + `27.5.1` (liveness probe).
+- venv: `~/.r9v-venv` (huggingface_hub for fetch/verify)
+
+## Verified at cutover (2026-09-27)
+
+- Setup: **PASS=28 WARN=3 FAIL=0** — 20/20 artifacts size+sha256 verified
+  (~93 GiB), PLE derived at exact pinned size, CED projector verified.
+- First-start doctor battery: **PASS=34 WARN=8 FAIL=0**, including
+  `runtime-worker-transport`: **pinned UVA reads, H2D copies, TP
+  all-reduce verified inside serving workers** on both BDFs.
+- Wire: `/v1/models` → `frank-primary`; `/health` + `/metrics` 200
+  (Prometheus scraping); "Say PONG" (non-thinking) → `'PONG'`, stop;
+  thinking requests populate `reasoning` (vLLM channel).
+- Memlock: the full tiered-expert load ran to healthy **under a plain dank
+  session** before the unit existed — the pin path does NOT hard-require
+  `LimitMEMLOCK=infinity` on this host, but the root unit keeps fleet
+  parity and headroom for the pinned reserve.
+- e2e gate: 32 passed / 1 failed — the failure is `11-admin open links`,
+  which needs FrankBook-Pro (192.168.0.112) awake; the machine was down and
+  WOL from both hosts failed. Suite minus that spec: **31 passed** (the
+  one-time 12-crawl flake re-passes in isolation).
+
+## Memory shape (measured)
+
+- Both cards ~29–30 GiB used of 31.86; engine budget
+  `CPU_OFFLOAD_GB_BY_DEVICE=146.25,78.75` (UVA host pin ≈ 43 GiB on rank 0
+  + 16 GiB PLE pinned reserve; PLE residency mode `ssd` mmap+trim).
+- `MemAvailable` floor: 56.3 GiB (`R9V_MIN_HOST_AVAILABLE_BYTES`).
+- Context envelope: 192,608 native window served as `MAX_MODEL_LEN=196608`;
+  harness `models.yml` carries `contextWindow: 196608`, `maxTokens: 57344`
+  (compaction-gap math: 70% × 196,608 = 137,625 + 57,344 ≤ 196,608).
+
+## Ops
+
+- **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve
+  script re-gates everything, relaunch through the shim, waits).
+- **First start after boot** re-warms ~93 GiB of page cache; expect several
+  minutes to /health; `TimeoutStartSec=1800`.
+- **Scratch/probe**: NEVER probe against the prod state dir. Copy it, set
+  `R9V_HOST_PORT` to a free port and `R9V_CONTAINER_NAME` to a scratch
+  name, and stop the probe before any deploy — a probe holding :7770
+  produces a false-green deploy (the 2026-09-26 lesson, now guarded by the
+  MainPID check).
+- **Qualification artifacts** (kept on-host):
+  `/home/dank/r9v-state/uncensored/qualification-*`,
+  failure bundles `failure-*` (review before sharing — raw logs).
+- **Models on the wire**: served alias `frank-primary`; thinking channel is
+  `reasoning` (vLLM), tool parser `qwen3_coder` (not qwen3_xml like the
+  vllm030 wiring had planned).
+
+## Known costs (accepted at selection)
+
+- **Abliterated weights** (Dyluhn uncensored build) — not the reference
+  model; answer-quality deltas are possible.
+- **192K envelope** — down from 262,144 on llama.cpp.
+- **Docker-pinned fork** — updates flow through image bundles + the shim,
+  never a bare container run; the launcher stays the single argv owner.
@@ -1,14 +1,14 @@
 # LLM Fleet — Knowledge Base

-Documentation for the LLM inference fleet on the two hosts (2026-09-26):
-**llama.cpp on all three endpoints** — Dankserver runs unsloth llama.cpp
-(the b11030-mix prebuilt, ROCm gfx120X) serving frank-primary
-(unsloth Qwen3.8-Flash-Next Q8_0 + the shared Q8_0 MTP head + ngram-mod +
-vision) across both Radeon AI PRO R9700s on :7770, and Dankclaw runs two
-endpoints — stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large,
-unsloth Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the
-gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS
-Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771).
+Documentation for the LLM inference fleet on the two hosts (2026-09-27):
+**Dankserver runs R9V radiance-vLLM on frank-primary** (the drwolfen fork
+serving the abliterated Dyluhn Qwen3.8-Flash-Next-Uncensored IQ4_XS + MTP4
+across both Radeon AI PRO R9700s on :7770, docker-pinned runtime driven
+through a docker→podman shim), and Dankclaw runs two llama.cpp endpoints —
+stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth
+Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151
+iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B
+QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771).

 > 2026-08-22: the former `vllmScripts` repository is merged into the
 > serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -22,10 +22,11 @@ graph LR
    subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
        A[2x Radeon AI PRO R9700
 gfx1201, 32 GiB each]
-        B2[unsloth llama.cpp b11030-mix, ROCm gfx120X
+        B2[R9V radiance-vLLM fork, docker-pinned
+via docker-podman shim
 frank-primary, port 7770]
-        C[Qwen3.8-Flash-Next Q8_0 + shared-Q8_0 MTP head
-layer split on both cards, 262144 ctx / 1 slot, ngram-mod, vision]
+        C[Qwen3.8-Flash-Next-Uncensored IQ4_XS
+TP=2 tiered experts, MTP4, CED, 196608 ctx, vision]
    end

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
@@ -57,8 +58,19 @@ frank-small, port 7771]

 ## Active Deployment

- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — the ACTIVE
-  frank-primary runbook (2026-09-26, root systemd unit, port 7770):
+- [[frank-primary-r9v-radiance-qwen38-uncensored]] — the ACTIVE
+  frank-primary runbook (2026-09-27, root systemd unit, port 7770): the
+  R9V radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned
+  image bundle `2dac17a2`, driven through a docker→podman shim) serving
+  Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
+  blockfp8 MTP4 (depth 4) + F16 vision on BOTH R9700s — TP=2 tiered
+  experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428
+  static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx, reasoning
+  channel `reasoning`, parser `qwen3_coder`. Root unit + LimitMEMLOCK;
+  the tuned 104-key shape lives in the on-host setup state. Rollback:
+  [[frank-primary-llamacpp-qwen38-flash-next-q8]]
+- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
+  (rollback target; the llama.cpp b11030 endpoint of 2026-09-26):
  unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime)
  serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s —
  routed experts and the 51B-param n-gram table computed in host RAM
@@ -124,8 +124,15 @@ The LLM inference fleet on the two hosts. Full index:

 - [[deployment]] — unified deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — the ACTIVE
-  frank-primary runbook: unsloth llama.cpp (b11030-mix prebuilt, ROCm
+- [[frank-primary-r9v-radiance-qwen38-uncensored]] — the ACTIVE
+  frank-primary runbook: the R9V radiance-vLLM fork (docker-pinned bundle
+  `2dac17a2`, docker→podman shim) serving the abliterated
+  Dyluhn Qwen3.8-Flash-Next-Uncensored IQ4_XS (92.39 GiB) + MTP4 + F16
+  vision on Dankserver's both R9700s — TP=2 tiered experts with a UVA
+  pinned host copy, PLE on ssd-mmap, CED on, 196608 ctx, root unit with
+  LimitMEMLOCK, the on-host 104-key tuned state, cutover and rollback
+- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
+  (rollback target): unsloth llama.cpp (b11030-mix prebuilt, ROCm
  gfx120X, root unit) serving unsloth Qwen3.8-Flash-Next at Q8_0 (175.3
  GiB) on Dankserver's both R9700s — the shared Q8_0 MTP head chained
  behind ngram-mod, BF16 vision, the experts and the 51B n-gram table