diff --git a/AGENTS.md b/AGENTS.md index b6730b0..7eeb41e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 GGUF shards) with the blockfp8 MTP4 head (depth 4) and F16 mmproj, TP=2 tiered experts (UVA pinned host copy 43 GiB rank 0 + 16 GiB PLE pinned reserve; PLE n-gram `ssd` mmap; budgets 146.25/78.75 GiB by device), CED on, 196608 ctx single stream, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 | +| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 | | Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), q8_0 KV, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | @@ -345,29 +345,40 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`. Rollback: [[frank-small-qwen38-gsq-rco]]. -### frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (Dankserver, ACTIVE) +### frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0 (Dankserver, ACTIVE) The **R9V radiance-vLLM fork** (`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt image bundle `r9v-qwen38-flash-next-wmma@sha256:2dac17a2…`) serving -`Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS` (abliterated weights, -92.39 GiB, 3 GGUF shards) with the blockfp8 MTP4 head (depth 4) and F16 -mmproj on **both** Radeon AI PRO R9700s (gfx1201, BDFs `0000:03:00.0` / -`0000:07:00.0`), one stream at **196,608 ctx**, **TP=2 tiered experts** -(UVA pinned host copy ~43 GiB on rank 0 + 16 GiB PLE pinned reserve; PLE -n-gram table on `ssd` mmap residency; per-device offload budgets -146.25/78.75 GiB), **CED on** (split-16 projector engagement at >=8192-token -prompts), tool parser `qwen3_coder`, reasoning channel `reasoning`. Cut -over 2026-09-27 from the llama.cpp b11030 endpoint (downtime ~1 h 24 m) at -Frank's direction. +the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** (175.30 GiB, 6 GGUF shards) +with the blockfp8 MTP4 head (depth 4) and the mmproj on **both** Radeon AI +PRO R9700s (gfx1201, BDFs `0000:03:00.0` / `0000:07:00.0`), one stream at +**131,072 ctx**, **TP=2 tiered experts** (80-slot LRU expert cache; PLE +n-gram table PINNED in host RAM, 54.4 GB), the **Qwen-Sharp v22.4.0 +template** (via the launcher's `R9V_CHAT_TEMPLATE_FILE` passthrough, fork +`30d4a32`), tool parser `qwen3_coder`, reasoning channel `reasoning`. Cut +over 2026-09-27 from the uncensored IQ4_XS payload (itself cut over from +llama.cpp b11030 the same day) at Frank's direction. -**The tuned shape lives on-host, not in the repo.** The profile's setup run -qualified a 104-key `R9V_*` config (placement, budgets, parsers, CED) into -`/home/dank/r9v-state/uncensored/setup.json` (`ready: true`); the repo env +**The tuned shape lives on-host, not in the repo.** The q8 profile's setup +run qualified the `R9V_*` config into +`/home/dank/r9v-state/q8xl/setup.json` (`ready: true`); the repo env file `scripts/frank-primary.env` carries ONLY production overrides (`R9V_HOST_BIND=0.0.0.0`, `R9V_HOST_PORT=7770`, `R9V_SERVED_MODEL_NAME=frank-primary`, `R9V_CONTAINER_NAME=frank-primary-r9v`, -`R9V_EXPECTED_GPU_BDFS`). Never copy tuned values from the state into the -repo — it would drift from the qualification. +`R9V_EXPECTED_GPU_BDFS`) plus two q8-only keys: the dev-overlay adapter +(`R9V_DEV_QWEN4_EXP_ADAPTER_PY`, fork commit `363d606` — the Q8_0 payload +quantizes BOTH fused hyper-connection shards, so the shard-uniformity +check needs the in-stream dequant) and `R9V_FULL_MUTABLE_CACHE=0` (the +overlay's env.always injects the uncensored 160-slot cache geometry, which +the q8 80-slot tiered cache rejects). Never copy tuned values from the +state into the repo — it would drift from the qualification. + +**Spec decoding is MTP4 only.** The fork's `speculative_config` is a +single dict — one method per engine, NO chaining — so the llama.cpp +`draft-mtp,ngram-mod` pattern is NOT expressible here. MTP4 is wired +(`R9V_MTP_SPEC_TOKENS=4`, draft TP=2); the PLE n-gram rides along +architecturally (`ngram_embedding`); `ngram_gpu` exists as an alternative +single method, not a chainable one. **Serving = the fork's own launcher through a docker→podman shim.** The R9V tooling is docker-pinned; the shim at @@ -383,9 +394,9 @@ and normalizes inspect output; (3) podman lacks docker's `SecurityOptions` /`ServerVersion` info fields — the shim answers them. **Unit and self-healing.** Root systemd unit (`scripts/frank-primary.service`, -`User=dank`, `LimitMEMLOCK=infinity`) — the tiered-expert UVA host copy pins -~43 GiB and the PLE reserve 16 GiB, and a root unit grants the pin budget -via PID1 (a rootless quadlet would be capped by the user manager's limit). +`User=dank`, `LimitMEMLOCK=infinity`) — the PLE pinned residency alone is +54.4 GB, and a root unit grants the pin budget via PID1 (a rootless +quadlet would be capped by the user manager's limit). The serve script (`scripts/serve-frank-primary.sh`) gates the setup state (ready + paths), the pinned image (RepoDigests), the PLE byte count (`stat -c %s`), every GGUF shard + the mmproj and the GPU device nodes, @@ -398,19 +409,22 @@ PLE/shards, `podman rm -f`s the container, waits out :7770, relinks/enables/ restarts the unit and health-gates with the MainPID check (the false-green stray law). -Verified at cutover: setup PASS=28/WARN=3/FAIL=0 (20/20 artifacts -sha256-verified, PLE at pinned size); first-start doctor battery PASS=34/ -WARN=8/FAIL=0 including pinned-UVA reads, H2D copies and the TP all-reduce -verified INSIDE serving workers; wire: alias `frank-primary`, `/health` + -`/metrics` 200, non-thinking PONG, thinking → `reasoning`, tool_calls -round-trip, vision ('red' for a red PNG), needle recall exact at 111,647 -prompt tokens (95% depth, ~4.5k tok/s effective prefill), decode 59–96 -tok/s (draft-acceptance dependent). MemAvailable floor 56.3 GiB. +Verified at cutover (2026-09-27): q8 setup state `ready: true`, all 20 +payload artifacts sha256-verified (175.30 GiB); the scratch-port bring-up +battery answered vision ("Red" for a 64×64 red PNG), MTP4 mean acceptance +length 3.19–5.00, and the PLE pinned proof (`mapping_bytes = +mapping_rss_bytes = 54,400,261,120`, fully RSS-resident); wire + tool +round-trip + template probe re-verified on :7770 after the deploy. No +throughput bench was run (Frank declined); the probe-era 11 tok/s decode +was residency-bound (halved expert headroom at double the bytes/expert), +not the production number. -Runbook: `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-uncensored.md`. -Rollback: the llama.cpp trio at commit `4992e5a` (unit/env/serve) and the -GGUF payload on disk — see [[frank-primary-llamacpp-qwen38-flash-next-q8]]; -a rollback is a redeploy. +Runbook: `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md`. +Rollback: the uncensored payload — flip `R9V_STATE_FILE` back to +`/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys +and the template keys, redeploy (state + `restore-config-20260927.env` + +payload intact on disk) — see +[[frank-primary-r9v-radiance-qwen38-uncensored]]. ## Key Lessons Learned Durable operational lessons from the fleet's history. Full detail in the @@ -759,7 +773,7 @@ Active: |---------|---------| | frank-large (stock llama.cpp Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md` | | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` | -| frank-primary (llama.cpp Qwen3.8-Flash-Next Q8_0 + shared MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md` | +| frank-primary (R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | @@ -767,10 +781,12 @@ Retired (rollback targets and history): | Runtime | Host | Window | Runbook | |---------|------|--------|---------| -| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` | +| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` | +| llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` | | ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` | | vllm-radiance TP=2 MXFP4 + DFlash2 (`frank-medium`) | Dankserver | 2026-09-20 → 09-26 | `frank-medium-radiance-mxfp4.md` | | Lucebox DS4.1 DeepSeek-V4.1-Flash ROCmFP23 (`frank-primary`) | Dankserver | 2026-09-26 (same-day swap) | `frank-primary-lucebox-ds4.md` | +| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` | | halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` | | CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` | | unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` | diff --git a/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md b/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md new file mode 100644 index 0000000..ae5d8f6 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md @@ -0,0 +1,177 @@ +--- +title: frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0 +service: frank-primary +host: dankserver +port: 7770 +status: active +deployed: 2026-09-27 +supersedes: "[[frank-primary-r9v-radiance-qwen38-uncensored]]" +tags: [llm, frank-primary, r9v, vllm, radiance, gfx1201, q8] +--- + +# frank-primary — R9V radiance-vLLM Q8_0 (dankserver, ACTIVE) + +The Dankserver LLM endpoint is the **R9V radiance-vLLM fork** +(`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt image bundle +`2dac17a2`) serving the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** payload +(175.30 GiB, 6 GGUF shards, sha256-verified per part) with the blockfp8 +MTP4 head and the mmproj, on **both R9700s** (gfx1201, BDFs `0000:03:00.0` +/ `0000:07:00.0`) as `frank-primary` on **:7770**. + +Cut over from the uncensored IQ4_XS payload on 2026-09-27 at Frank's +direction. Rollback target: [[frank-primary-r9v-radiance-qwen38-uncensored]] +— the uncensored state dir (`/home/dank/r9v-state/uncensored/`, `ready: +true`), its frozen config (`restore-config-20260927.env`) and payload stay +on disk; a rollback is an env-file `R9V_STATE_FILE` flip plus a redeploy. + +## Why the swap + +The uncensored IQ4_XS payload measured 59–96 tok/s decode with +draft-acceptance swings; the Q8_0 payload trades VRAM-resident expert +headroom for precision — quantization to Q8_0 doubles expert bytes per +block, so the tiered-expert residency budget halves (see [[live-llm-benchmark-methodology]] +caveats on decode-vs-residency). The probe-era numbers are **not** the +cutover numbers; see "Known costs" below. + +## Production shape (repo-tracked) + +The repo carries a **thin** wiring; the engine shape lives on-host: + +| Piece | File | Law | +|---|---|---| +| Unit | `scripts/frank-primary.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-primary-r9v`, `Restart=on-failure` | +| Env | `scripts/frank-primary.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary`, container `frank-primary-r9v`, GPU BDFs, dev-overlay adapter, `R9V_FULL_MUTABLE_CACHE=0`, sharp chat template | +| Serve | `scripts/serve-frank-primary.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` | +| Deployer | `scripts/deploy-remote.sh` `deploy_llm_primary` | Gates on setup state ready, image digest, PLE bytes, all six shard files + mmproj; MainPID false-green check | + +**Why the tuned shape lives on-host:** the q8 profile's setup run +qualified the `R9V_*` config (tiered-expert placement, per-rank offload +budgets, 80-slot tiered cache, PLE **pinned** residency 54.4 GB, MTP4, +parser wiring, `MAX_MODEL_LEN=131072`). It is written to +`/home/dank/r9v-state/q8xl/setup.json` and the serve script layers it +UNDER the repo env (repo keys win). Copying those values into the repo +would drift from the qualification — don't. + +## Spec decode: MTP4 only — the ngram-mod ask is NOT expressible + +llama.cpp chains drafter types (`--spec-type draft-mtp,ngram-mod` on +frank-small/large). The fork's `speculative_config` is a **single dict** +(one method per engine): `mtp`, `ngram`, `ngram_gpu`, `dflash`, … are +mutually exclusive — no chaining seam exists in the fork. What IS active: + +- **MTP4** (`R9V_MTP_SPEC_TOKENS=4`, blockfp8 draft head, draft TP=2) — + measured mean acceptance length 3.19–5.00 during qualification. +- **The PLE n-gram** — architectural (`ngram_embedding` module), rides + along automatically; it is not the llama.cpp `ngram-mod` matcher and + cannot be "added" as a second spec method. +- `ngram_gpu` exists as an ALTERNATIVE single method (fork's GPU n-gram + proposer) — swapping MTP4 for it is an either/or, not a chain. + +## Sharp chat template (fleet parity with frank-large) + +`scripts/qwen-sharp-v22.4.0.jinja` is served via the launcher's opt-in +`R9V_CHAT_TEMPLATE_FILE` passthrough (fork commit `30d4a32`): the file is +mounted read-only at `/opt/r9v/share/chat-template.jinja` and passed as +`--chat-template`, which wins over the tokenizer dir's +`metadata/chat_template.jinja`; `R9V_CHAT_TEMPLATE_KWARGS` carries +`--default-chat-template-kwargs {"preserve_thinking":true}` (the template +defaults `preserve_thinking` true natively; request-level kwargs still win +per request). The template folds reasoning efforts (high/xhigh/max → +xhigh, minimal/low → low), matching `models.yml`'s `reasoningEffortMap`. +The sha-pinned `metadata/chat_template.jinja` in the package is NEVER +edited in place — mutating it drifts the qualification. + +## Two q8-only runtime laws + +1. **Quantized inject shard** — the Q8_0 payload quantizes BOTH fused + hyper-connection shards (the IQ4_XS package ships inject as F32), so + the shard-uniformity check rejects the fused projection at load. + `R9V_DEV_QWEN4_EXP_ADAPTER_PY` (fork commit `363d606`) mounts a + dev-overlay adapter that dequantizes a quantized inject shard + in-stream; uncensored payloads pass through unchanged. +2. **`R9V_FULL_MUTABLE_CACHE=0`** — the runtime overlay's `env.always` + injects the uncensored 160-slot full-mutable cache geometry, which the + q8 profile's 80-slot tiered cache fails with a baseline mismatch. The + launcher passes this key AFTER the overlay envs (docker last-wins). + +## Fork checkout + shim (on-host, outside the repo) + +- Checkout: `/home/dank/GitPackages/radiance-vllm-qwen4exp` (fork commits + `363d606` dev-overlay, `2fe6dbb`, `50a79ec` qualification doc, + `30d4a32` chat-template passthrough — local-only, remote is GitHub) +- State: `/home/dank/r9v-state/q8xl/setup.json` (0600, `ready: true`) +- Model dir: `/home/dank/models/Qwen3.8-Flash-Next-Q8_0-R9V/` — 175.30 GiB + payload as six `Q8_0/Qwen3.8-Flash-Next-Q8_0-0000N-of-00006.gguf` shards + + mmproj; PLE derived at the pinned size 54,400,261,120 B +- Docker→podman shim: `/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker` + (PATH-prepended by the serve script). The R9V tooling is docker-pinned; + the shim carries three laws: + 1. **Image ID law** — the bundle is a containerd/OCI docker-save; docker + keys the image by its OCI MANIFEST digest (`sha256:2dac17a2…`), podman + by config digest but preserves the manifest digest in **RepoDigests**. + The shim answers `image inspect sha256:X` from RepoDigests and rewrites + bare `sha256:X` run refs to `repo@sha256:X`. + 2. **LogConfig law** — podman inspect answers + `HostConfig.LogConfig.Config: null` where docker returns the log-opts + map (the doctor crashes on `.get("Config", {}) == null`). The shim + records `run --log-opt` pairs and normalizes inspect output. + 3. **docker info law** — podman lacks `SecurityOptions`/`ServerVersion`; + the shim answers rootless + `27.5.1` (liveness probe). +- venv: `~/.r9v-venv` (huggingface_hub for fetch/verify) + +## Bring-up battery (qualification, 2026-09-27) + +- Setup `ready: true`; all 20 payload artifacts size+sha256 verified + (175.30 GiB total across the six shards). +- Vision: 64×64 red PNG → "Red" (79 tok, 7.4 s end-to-end on the scratch + port). Prose: 100 tok in 9.1 s ≈ 11.0 tok/s wall **on a cold cache with + reasoning tokens included in the wall clock** — decode-rate context, not + a bench (no throughput bench was run, per Frank's direction). +- MTP4: mean acceptance length 3.19–5.00 across the battery. +- PLE pinned proof: `mapping_bytes = mapping_rss_bytes = + 54,400,261,120` (fully RSS-resident); doubling for re-pin would need + 108,800,522,240 ≤ available 140,876,808,192; registration took 446 s. + +## Memory shape (measured) + +- Load: 21.84 GiB / worker at ~158 s; cold start → serve ~16 min. +- PLE pinned residency 54.4 GB in host RAM (vs the uncensored payload's + ssd-mmap residency). +- Context envelope: `R9V_MAX_MODEL_LEN=131072`; harness `models.yml` + carries `contextWindow: 131072`, `maxTokens: 36864` (compaction-gap + math: 70% × 131,072 = 91,750 + 36,864 = 128,614 ≤ 131,072). +- Prefix caching OFF (`R9V_ENABLE_PREFIX_CACHING=0`), CACHE192 off — the + KV envelope at 131,072 is smaller than the uncensored 196,608 shape. + +## Ops + +- **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve + script re-gates everything, relaunch through the shim, waits). +- **First start after boot** re-warms ~175 GiB of page cache; expect + minutes to /health; `TimeoutStartSec=1800`. +- **Scratch/probe**: NEVER probe against the prod state dir. Copy it, set + `R9V_HOST_PORT` to a free port and `R9V_CONTAINER_NAME` to a scratch + name, and stop the probe before any deploy — a probe holding :7770 + produces a false-green deploy (the 2026-09-26 lesson, guarded by the + MainPID check). +- **Rollback to uncensored**: point + `scripts/frank-primary.env` `R9V_STATE_FILE` back at + `/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys + (adapter, FULL_MUTABLE_CACHE) and the template keys, redeploy — the + uncensored state dir and payload are intact on disk. +- **Models on the wire**: served alias `frank-primary`; thinking channel + is `reasoning` (vLLM); tool parser `qwen3_coder`. + +## Known costs (accepted at cutover) + +- **Residency-halved experts** — Q8_0 doubles expert bytes vs IQ4_XS, so + the tiered cache holds half as many experts in VRAM at the same budget; + the probe's 11 tok/s decode was residency-bound, and the production + number depends on the re-qualified budgets in the q8 state. +- **128K envelope** — down from the uncensored 196,608 (which itself was + down from 262,144 on llama.cpp). +- **Reference quant** — back to the unsloth Q8_0 weights (no + abliteration); answer-quality profile differs from the uncensored + incumbent by design. +- **Docker-pinned fork** — updates flow through image bundles + the shim, + never a bare container run; the launcher stays the single argv owner. diff --git a/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-uncensored.md b/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-uncensored.md index 8cba619..dc81d68 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-uncensored.md +++ b/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-uncensored.md @@ -3,13 +3,19 @@ title: frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS service: frank-primary host: dankserver port: 7770 -status: active -deployed: 2026-09-27 +status: retired +retired: 2026-09-27 +superseded_by: "[[frank-primary-r9v-radiance-qwen38-q8]]" supersedes: "[[frank-primary-llamacpp-qwen38-flash-next-q8]]" -tags: [llm, frank-primary, r9v, vllm, radiance, gfx1201] +tags: [llm, frank-primary, r9v, vllm, radiance, gfx1201, retired] --- -# frank-primary — R9V radiance-vLLM (dankserver, ACTIVE) +# frank-primary — R9V radiance-vLLM (dankserver, RETIRED) + +> **RETIRED 2026-09-27** — superseded the same day by the Q8_0 cutover, +> [[frank-primary-r9v-radiance-qwen38-q8]]. Kept as the rollback runbook: +> the uncensored state dir, frozen config and payload are intact on disk; +> the rollback procedure lives in that page's Ops section. The Dankserver LLM endpoint is the **R9V radiance-vLLM fork** (`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt) serving diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index f7b0afb..c7e4f25 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -2,9 +2,9 @@ Documentation for the LLM inference fleet on the two hosts (2026-09-27): **Dankserver runs R9V radiance-vLLM on frank-primary** (the drwolfen fork -serving the abliterated Dyluhn Qwen3.8-Flash-Next-Uncensored IQ4_XS + MTP4 -across both Radeon AI PRO R9700s on :7770, docker-pinned runtime driven -through a docker→podman shim), and Dankclaw runs two llama.cpp endpoints — +serving the unsloth Qwen3.8-Flash-Next Q8_0 + MTP4 across both Radeon AI +PRO R9700s on :7770, docker-pinned runtime driven through a docker→podman +shim, the Qwen-Sharp template, 131072 ctx), and Dankclaw runs two llama.cpp endpoints — stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B @@ -25,8 +25,8 @@ gfx1201, 32 GiB each] B2[R9V radiance-vLLM fork, docker-pinned via docker-podman shim frank-primary, port 7770] - C[Qwen3.8-Flash-Next-Uncensored IQ4_XS -TP=2 tiered experts, MTP4, CED, 196608 ctx, vision] + C[Qwen3.8-Flash-Next Q8_0 +TP=2 tiered experts, MTP4, sharp template, 131072 ctx, vision] end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] @@ -58,16 +58,27 @@ frank-small, port 7771] ## Active Deployment -- [[frank-primary-r9v-radiance-qwen38-uncensored]] — the ACTIVE - frank-primary runbook (2026-09-27, root systemd unit, port 7770): the - R9V radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned +- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary + runbook (2026-09-27, root systemd unit, port 7770): the R9V + radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned image bundle `2dac17a2`, driven through a docker→podman shim) serving + the unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) + + blockfp8 MTP4 (depth 4) + vision on BOTH R9700s — TP=2 tiered experts + (80-slot LRU cache), PLE n-gram PINNED in host RAM (54.4 GB), CED off, + 131072 ctx, the Qwen-Sharp v22.4.0 template via the launcher's + `R9V_CHAT_TEMPLATE_FILE` passthrough (fork `30d4a32`), the quantized + inject-shard dev-overlay adapter (fork `363d606`), reasoning channel + `reasoning`, parser `qwen3_coder`. The tuned shape lives in the + on-host q8xl setup state. Rollback: + [[frank-primary-r9v-radiance-qwen38-uncensored]] +- [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27 + (same-day swap to the Q8_0 payload; rollback target, state + payload + intact on disk): the R9V radiance-vLLM fork serving Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) + blockfp8 MTP4 (depth 4) + F16 vision on BOTH R9700s — TP=2 tiered experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428 - static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx, reasoning - channel `reasoning`, parser `qwen3_coder`. Root unit + LimitMEMLOCK; - the tuned 104-key shape lives in the on-host setup state. Rollback: + static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx. Cut over + from llama.cpp the same day; rollback target [[frank-primary-llamacpp-qwen38-flash-next-q8]] - [[frank-primary-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27 (rollback target; the llama.cpp b11030 endpoint of 2026-09-26): diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index c202c9e..59be5bd 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -1,6 +1,7 @@ # Bundle Change Log ## 2026-09-27 +- `/scripts/frank-primary.env`, `/scripts/frank-primary.service`, `/scripts/serve-frank-primary.sh`, `/scripts/deploy-remote.sh`, `/tests/frank_primary_unit.bats`, `/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` (new), AGENTS.md + LLM index — **frank-primary cut over to the Q8_0 payload** (uncensored IQ4_XS → unsloth Qwen3.8-Flash-Next Q8_0, 175.30 GiB / 6 shards) at Frank's direction. Launcher gained the opt-in `R9V_CHAT_TEMPLATE_FILE`/`R9V_CHAT_TEMPLATE_KWARGS` passthrough (fork commit `30d4a32`, local-only) — the Qwen-Sharp v22.4.0 template now serves, fleet parity with frank-large. Env points at `/home/dank/r9v-state/q8xl/setup.json` + the quantized-inject dev-overlay adapter (fork `363d606`) + `R9V_FULL_MUTABLE_CACHE=0`; shard gates extended to SHARD4/5/6 (unset-safe); seams recalibrated; ctx window 131072 (models.yml maxTokens 36864). Spec decode is MTP4 only — the fork's `speculative_config` is a single dict, no chaining, so llama.cpp's `draft-mtp,ngram-mod` pattern is NOT expressible (PLE n-gram rides along architecturally). Uncensored state + payload remain on disk as the rollback target. 346 tests green. - Dankserver frank-primary slot (experiment window) — the `radiance-vllm-qwen4exp` fork qualified the unsloth `Qwen3.8-Flash-Next-GGUF` **Q8_0** payload (175.30 GiB, 6 split parts) on the dual R9700. Two launch blockers fixed loader-side in the fork: `runtimes/dev-overrides/qwen4_exp.py` dequantizes quantized `block_inject` shards in-stream (the Q8_0 payload quantizes BOTH fused-HC shards; IQ4_XS ships inject as F32 — the uniformity check rejected the mixed projection) and `R9V_FULL_MUTABLE_CACHE=0` passthrough for the 80-slot tiered cache (fork commits 363d606, 2fe6dbb, 50a79ec — GitHub remote, not pushed). Battery on :7773: text + reasoning channel, vision ("Red" on a red PNG), prose ≈11.0 tok/s wall, MTP4 mean acceptance length 3.19–5.00; PLE pinned residency proven (`mapping_bytes=54400261120`, `mapping_rss_bytes` == mapping size, both ranks required 108.8 GiB vs 140.9 available). Relaunch recipe saved at `dankserver:/home/dank/r9v-state/q8xl/START-COMMAND.txt`; record at fork `docs/qualification/qwen38-q8-0-dual-r9700-2026-09-27.md`. - Dankserver frank-primary restore — uncensored UD-IQ4_XS back on `0.0.0.0:7770` as `frank-primary-r9v` from its frozen config (health 200 in ~60 s; LAN-path completion returns content + reasoning); `frank-primary.service` (llama.cpp) **runtime-masked** for the window (`/run/systemd/system/frank-primary.service → /dev/null` — the `/etc` relink symlink blocks a plain mask; self-clears at reboot, unit file untouched) and the container runs `restart=no`, so a reboot returns the repo-configured llama.cpp with no contention. Restore config preserved at `dankserver:/home/dank/r9v-state/uncensored/restore-config-20260927.env`; scratch in `dankserver:~/.AGENTTRASH/20260927-q8xl-bringup/`.