diff --git a/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-uncensored.md b/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-uncensored.md new file mode 100644 index 0000000..8cba619 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-uncensored.md @@ -0,0 +1,126 @@ +--- +title: frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS +service: frank-primary +host: dankserver +port: 7770 +status: active +deployed: 2026-09-27 +supersedes: "[[frank-primary-llamacpp-qwen38-flash-next-q8]]" +tags: [llm, frank-primary, r9v, vllm, radiance, gfx1201] +--- + +# frank-primary — R9V radiance-vLLM (dankserver, ACTIVE) + +The Dankserver LLM endpoint is the **R9V radiance-vLLM fork** +(`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt) serving +`Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS` — the abliterated +Qwen3.8-Flash-Next at IQ4_XS (92.39 GiB, 3 GGUF shards) with the blockfp8 +MTP4 head and the F16 mmproj — on **both R9700s** (gfx1201, BDFs +`0000:03:00.0` / `0000:07:00.0`) as `frank-primary` on **:7770**. + +Replaced llama.cpp b11030 (Qwen3.8-Flash-Next Q_8) on 2026-09-27 at Frank's +direction; downtime window ~1 h 20 min (killed 02:47, healthy 04:11 PDT). +Rollback target: [[frank-primary-llamacpp-qwen38-flash-next-q8]] — the +llama.cpp trio (unit/env/serve) is at commit `4992e5a`+ (the repo-side +wiring; host unit was the relinked one) and the GGUF payload stays on disk, +so a rollback is a redeploy. + +## Production shape (repo-tracked) + +The repo carries a **thin** wiring; the engine shape lives on-host: + +| Piece | File | Law | +|---|---|---| +| Unit | `scripts/frank-primary.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-primary-r9v`, `Restart=on-failure` | +| Env | `scripts/frank-primary.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary`, container `frank-primary-r9v`, GPU BDFs | +| Serve | `scripts/serve-frank-primary.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` | +| Deployer | `scripts/deploy-remote.sh` `deploy_llm_primary` | Gates on setup state ready, image digest, PLE bytes, shard files; MainPID false-green check | + +**Why the tuned shape lives on-host:** the profile's setup run qualified a +104-key `R9V_*` config (tiered-expert placement, per-rank offload budgets +146.25/78.75 GiB, CED on, MTP4, parser wiring, `MAX_MODEL_LEN=196608`). It +is written to `/home/dank/r9v-state/uncensored/setup.json` and the serve +script layers it UNDER the repo env (repo keys win). Copying those values +into the repo would drift from the qualification — don't. + +**Self-healing law:** the launcher `--detach`es; the serve script therefore +blocks on `docker wait`. Without the wait, a crashed engine is a green +unit. `docker wait` cannot stop the container child, hence `ExecStop`. + +## Fork checkout + shim (on-host, outside the repo) + +- Checkout: `/home/dank/GitPackages/radiance-vllm-qwen4exp` +- State: `/home/dank/r9v-state/uncensored/setup.json` (0600, `ready: true`) +- Model dir: `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS/` + (92.39 GiB payload + `r9v-data/per_layer_token_embd.iq4_nl.bin` 26.82 GiB + PLE, sha256 `34fa36f8…`) +- Docker→podman shim: `/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker` + (PATH-prepended by the serve script). The R9V tooling is docker-pinned; + the shim carries three laws: + 1. **Image ID law** — the bundle is a containerd/OCI docker-save; docker + keys the image by its OCI MANIFEST digest (`sha256:2dac17a2…`), podman + by config digest but preserves the manifest digest in **RepoDigests**. + The shim answers `image inspect sha256:X` from RepoDigests and rewrites + bare `sha256:X` run refs to `repo@sha256:X`. + 2. **LogConfig law** — podman inspect answers + `HostConfig.LogConfig.Config: null` where docker returns the log-opts + map (the doctor crashes on `.get("Config", {}) == null`). The shim + records `run --log-opt` pairs and normalizes inspect output. + 3. **docker info law** — podman lacks `SecurityOptions`/`ServerVersion`; + the shim answers rootless + `27.5.1` (liveness probe). +- venv: `~/.r9v-venv` (huggingface_hub for fetch/verify) + +## Verified at cutover (2026-09-27) + +- Setup: **PASS=28 WARN=3 FAIL=0** — 20/20 artifacts size+sha256 verified + (~93 GiB), PLE derived at exact pinned size, CED projector verified. +- First-start doctor battery: **PASS=34 WARN=8 FAIL=0**, including + `runtime-worker-transport`: **pinned UVA reads, H2D copies, TP + all-reduce verified inside serving workers** on both BDFs. +- Wire: `/v1/models` → `frank-primary`; `/health` + `/metrics` 200 + (Prometheus scraping); "Say PONG" (non-thinking) → `'PONG'`, stop; + thinking requests populate `reasoning` (vLLM channel). +- Memlock: the full tiered-expert load ran to healthy **under a plain dank + session** before the unit existed — the pin path does NOT hard-require + `LimitMEMLOCK=infinity` on this host, but the root unit keeps fleet + parity and headroom for the pinned reserve. +- e2e gate: 32 passed / 1 failed — the failure is `11-admin open links`, + which needs FrankBook-Pro (192.168.0.112) awake; the machine was down and + WOL from both hosts failed. Suite minus that spec: **31 passed** (the + one-time 12-crawl flake re-passes in isolation). + +## Memory shape (measured) + +- Both cards ~29–30 GiB used of 31.86; engine budget + `CPU_OFFLOAD_GB_BY_DEVICE=146.25,78.75` (UVA host pin ≈ 43 GiB on rank 0 + + 16 GiB PLE pinned reserve; PLE residency mode `ssd` mmap+trim). +- `MemAvailable` floor: 56.3 GiB (`R9V_MIN_HOST_AVAILABLE_BYTES`). +- Context envelope: 192,608 native window served as `MAX_MODEL_LEN=196608`; + harness `models.yml` carries `contextWindow: 196608`, `maxTokens: 57344` + (compaction-gap math: 70% × 196,608 = 137,625 + 57,344 ≤ 196,608). + +## Ops + +- **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve + script re-gates everything, relaunch through the shim, waits). +- **First start after boot** re-warms ~93 GiB of page cache; expect several + minutes to /health; `TimeoutStartSec=1800`. +- **Scratch/probe**: NEVER probe against the prod state dir. Copy it, set + `R9V_HOST_PORT` to a free port and `R9V_CONTAINER_NAME` to a scratch + name, and stop the probe before any deploy — a probe holding :7770 + produces a false-green deploy (the 2026-09-26 lesson, now guarded by the + MainPID check). +- **Qualification artifacts** (kept on-host): + `/home/dank/r9v-state/uncensored/qualification-*`, + failure bundles `failure-*` (review before sharing — raw logs). +- **Models on the wire**: served alias `frank-primary`; thinking channel is + `reasoning` (vLLM), tool parser `qwen3_coder` (not qwen3_xml like the + vllm030 wiring had planned). + +## Known costs (accepted at selection) + +- **Abliterated weights** (Dyluhn uncensored build) — not the reference + model; answer-quality deltas are possible. +- **192K envelope** — down from 262,144 on llama.cpp. +- **Docker-pinned fork** — updates flow through image bundles + the shim, + never a bare container run; the launcher stays the single argv owner. diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 950e130..f7b0afb 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -1,14 +1,14 @@ # LLM Fleet — Knowledge Base -Documentation for the LLM inference fleet on the two hosts (2026-09-26): -**llama.cpp on all three endpoints** — Dankserver runs unsloth llama.cpp -(the b11030-mix prebuilt, ROCm gfx120X) serving frank-primary -(unsloth Qwen3.8-Flash-Next Q8_0 + the shared Q8_0 MTP head + ngram-mod + -vision) across both Radeon AI PRO R9700s on :7770, and Dankclaw runs two -endpoints — stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, -unsloth Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the -gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS -Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771). +Documentation for the LLM inference fleet on the two hosts (2026-09-27): +**Dankserver runs R9V radiance-vLLM on frank-primary** (the drwolfen fork +serving the abliterated Dyluhn Qwen3.8-Flash-Next-Uncensored IQ4_XS + MTP4 +across both Radeon AI PRO R9700s on :7770, docker-pinned runtime driven +through a docker→podman shim), and Dankclaw runs two llama.cpp endpoints — +stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth +Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151 +iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B +QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771). > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -22,10 +22,11 @@ graph LR subgraph Dankserver[AMD Dankserver - Ubuntu 26.04] A[2x Radeon AI PRO R9700 gfx1201, 32 GiB each] - B2[unsloth llama.cpp b11030-mix, ROCm gfx120X + B2[R9V radiance-vLLM fork, docker-pinned +via docker-podman shim frank-primary, port 7770] - C[Qwen3.8-Flash-Next Q8_0 + shared-Q8_0 MTP head -layer split on both cards, 262144 ctx / 1 slot, ngram-mod, vision] + C[Qwen3.8-Flash-Next-Uncensored IQ4_XS +TP=2 tiered experts, MTP4, CED, 196608 ctx, vision] end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] @@ -57,8 +58,19 @@ frank-small, port 7771] ## Active Deployment -- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — the ACTIVE - frank-primary runbook (2026-09-26, root systemd unit, port 7770): +- [[frank-primary-r9v-radiance-qwen38-uncensored]] — the ACTIVE + frank-primary runbook (2026-09-27, root systemd unit, port 7770): the + R9V radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned + image bundle `2dac17a2`, driven through a docker→podman shim) serving + Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) + + blockfp8 MTP4 (depth 4) + F16 vision on BOTH R9700s — TP=2 tiered + experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428 + static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx, reasoning + channel `reasoning`, parser `qwen3_coder`. Root unit + LimitMEMLOCK; + the tuned 104-key shape lives in the on-host setup state. Rollback: + [[frank-primary-llamacpp-qwen38-flash-next-q8]] +- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27 + (rollback target; the llama.cpp b11030 endpoint of 2026-09-26): unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s — routed experts and the 51B-param n-gram table computed in host RAM diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 2d62143..31f1901 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -124,8 +124,15 @@ The LLM inference fleet on the two hosts. Full index: - [[deployment]] — unified deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — the ACTIVE - frank-primary runbook: unsloth llama.cpp (b11030-mix prebuilt, ROCm +- [[frank-primary-r9v-radiance-qwen38-uncensored]] — the ACTIVE + frank-primary runbook: the R9V radiance-vLLM fork (docker-pinned bundle + `2dac17a2`, docker→podman shim) serving the abliterated + Dyluhn Qwen3.8-Flash-Next-Uncensored IQ4_XS (92.39 GiB) + MTP4 + F16 + vision on Dankserver's both R9700s — TP=2 tiered experts with a UVA + pinned host copy, PLE on ssd-mmap, CED on, 196608 ctx, root unit with + LimitMEMLOCK, the on-host 104-key tuned state, cutover and rollback +- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27 + (rollback target): unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, root unit) serving unsloth Qwen3.8-Flash-Next at Q8_0 (175.3 GiB) on Dankserver's both R9700s — the shared Q8_0 MTP head chained behind ngram-mod, BF16 vision, the experts and the 51B n-gram table