docs: frank-primary q8 cutover — new runbook, uncensored retired as rollback, fleet table/index/log synced
This commit is contained in:
Frank
@@ -0,0 +1,177 @@
|
||||
---
|
||||
title: frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0
|
||||
service: frank-primary
|
||||
host: dankserver
|
||||
port: 7770
|
||||
status: active
|
||||
deployed: 2026-09-27
|
||||
supersedes: "[[frank-primary-r9v-radiance-qwen38-uncensored]]"
|
||||
tags: [llm, frank-primary, r9v, vllm, radiance, gfx1201, q8]
|
||||
---
|
||||
|
||||
# frank-primary — R9V radiance-vLLM Q8_0 (dankserver, ACTIVE)
|
||||
|
||||
The Dankserver LLM endpoint is the **R9V radiance-vLLM fork**
|
||||
(`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt image bundle
|
||||
`2dac17a2`) serving the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** payload
|
||||
(175.30 GiB, 6 GGUF shards, sha256-verified per part) with the blockfp8
|
||||
MTP4 head and the mmproj, on **both R9700s** (gfx1201, BDFs `0000:03:00.0`
|
||||
/ `0000:07:00.0`) as `frank-primary` on **:7770**.
|
||||
|
||||
Cut over from the uncensored IQ4_XS payload on 2026-09-27 at Frank's
|
||||
direction. Rollback target: [[frank-primary-r9v-radiance-qwen38-uncensored]]
|
||||
— the uncensored state dir (`/home/dank/r9v-state/uncensored/`, `ready:
|
||||
true`), its frozen config (`restore-config-20260927.env`) and payload stay
|
||||
on disk; a rollback is an env-file `R9V_STATE_FILE` flip plus a redeploy.
|
||||
|
||||
## Why the swap
|
||||
|
||||
The uncensored IQ4_XS payload measured 59–96 tok/s decode with
|
||||
draft-acceptance swings; the Q8_0 payload trades VRAM-resident expert
|
||||
headroom for precision — quantization to Q8_0 doubles expert bytes per
|
||||
block, so the tiered-expert residency budget halves (see [[live-llm-benchmark-methodology]]
|
||||
caveats on decode-vs-residency). The probe-era numbers are **not** the
|
||||
cutover numbers; see "Known costs" below.
|
||||
|
||||
## Production shape (repo-tracked)
|
||||
|
||||
The repo carries a **thin** wiring; the engine shape lives on-host:
|
||||
|
||||
| Piece | File | Law |
|
||||
|---|---|---|
|
||||
| Unit | `scripts/frank-primary.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-primary-r9v`, `Restart=on-failure` |
|
||||
| Env | `scripts/frank-primary.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary`, container `frank-primary-r9v`, GPU BDFs, dev-overlay adapter, `R9V_FULL_MUTABLE_CACHE=0`, sharp chat template |
|
||||
| Serve | `scripts/serve-frank-primary.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
|
||||
| Deployer | `scripts/deploy-remote.sh` `deploy_llm_primary` | Gates on setup state ready, image digest, PLE bytes, all six shard files + mmproj; MainPID false-green check |
|
||||
|
||||
**Why the tuned shape lives on-host:** the q8 profile's setup run
|
||||
qualified the `R9V_*` config (tiered-expert placement, per-rank offload
|
||||
budgets, 80-slot tiered cache, PLE **pinned** residency 54.4 GB, MTP4,
|
||||
parser wiring, `MAX_MODEL_LEN=131072`). It is written to
|
||||
`/home/dank/r9v-state/q8xl/setup.json` and the serve script layers it
|
||||
UNDER the repo env (repo keys win). Copying those values into the repo
|
||||
would drift from the qualification — don't.
|
||||
|
||||
## Spec decode: MTP4 only — the ngram-mod ask is NOT expressible
|
||||
|
||||
llama.cpp chains drafter types (`--spec-type draft-mtp,ngram-mod` on
|
||||
frank-small/large). The fork's `speculative_config` is a **single dict**
|
||||
(one method per engine): `mtp`, `ngram`, `ngram_gpu`, `dflash`, … are
|
||||
mutually exclusive — no chaining seam exists in the fork. What IS active:
|
||||
|
||||
- **MTP4** (`R9V_MTP_SPEC_TOKENS=4`, blockfp8 draft head, draft TP=2) —
|
||||
measured mean acceptance length 3.19–5.00 during qualification.
|
||||
- **The PLE n-gram** — architectural (`ngram_embedding` module), rides
|
||||
along automatically; it is not the llama.cpp `ngram-mod` matcher and
|
||||
cannot be "added" as a second spec method.
|
||||
- `ngram_gpu` exists as an ALTERNATIVE single method (fork's GPU n-gram
|
||||
proposer) — swapping MTP4 for it is an either/or, not a chain.
|
||||
|
||||
## Sharp chat template (fleet parity with frank-large)
|
||||
|
||||
`scripts/qwen-sharp-v22.4.0.jinja` is served via the launcher's opt-in
|
||||
`R9V_CHAT_TEMPLATE_FILE` passthrough (fork commit `30d4a32`): the file is
|
||||
mounted read-only at `/opt/r9v/share/chat-template.jinja` and passed as
|
||||
`--chat-template`, which wins over the tokenizer dir's
|
||||
`metadata/chat_template.jinja`; `R9V_CHAT_TEMPLATE_KWARGS` carries
|
||||
`--default-chat-template-kwargs {"preserve_thinking":true}` (the template
|
||||
defaults `preserve_thinking` true natively; request-level kwargs still win
|
||||
per request). The template folds reasoning efforts (high/xhigh/max →
|
||||
xhigh, minimal/low → low), matching `models.yml`'s `reasoningEffortMap`.
|
||||
The sha-pinned `metadata/chat_template.jinja` in the package is NEVER
|
||||
edited in place — mutating it drifts the qualification.
|
||||
|
||||
## Two q8-only runtime laws
|
||||
|
||||
1. **Quantized inject shard** — the Q8_0 payload quantizes BOTH fused
|
||||
hyper-connection shards (the IQ4_XS package ships inject as F32), so
|
||||
the shard-uniformity check rejects the fused projection at load.
|
||||
`R9V_DEV_QWEN4_EXP_ADAPTER_PY` (fork commit `363d606`) mounts a
|
||||
dev-overlay adapter that dequantizes a quantized inject shard
|
||||
in-stream; uncensored payloads pass through unchanged.
|
||||
2. **`R9V_FULL_MUTABLE_CACHE=0`** — the runtime overlay's `env.always`
|
||||
injects the uncensored 160-slot full-mutable cache geometry, which the
|
||||
q8 profile's 80-slot tiered cache fails with a baseline mismatch. The
|
||||
launcher passes this key AFTER the overlay envs (docker last-wins).
|
||||
|
||||
## Fork checkout + shim (on-host, outside the repo)
|
||||
|
||||
- Checkout: `/home/dank/GitPackages/radiance-vllm-qwen4exp` (fork commits
|
||||
`363d606` dev-overlay, `2fe6dbb`, `50a79ec` qualification doc,
|
||||
`30d4a32` chat-template passthrough — local-only, remote is GitHub)
|
||||
- State: `/home/dank/r9v-state/q8xl/setup.json` (0600, `ready: true`)
|
||||
- Model dir: `/home/dank/models/Qwen3.8-Flash-Next-Q8_0-R9V/` — 175.30 GiB
|
||||
payload as six `Q8_0/Qwen3.8-Flash-Next-Q8_0-0000N-of-00006.gguf` shards
|
||||
+ mmproj; PLE derived at the pinned size 54,400,261,120 B
|
||||
- Docker→podman shim: `/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker`
|
||||
(PATH-prepended by the serve script). The R9V tooling is docker-pinned;
|
||||
the shim carries three laws:
|
||||
1. **Image ID law** — the bundle is a containerd/OCI docker-save; docker
|
||||
keys the image by its OCI MANIFEST digest (`sha256:2dac17a2…`), podman
|
||||
by config digest but preserves the manifest digest in **RepoDigests**.
|
||||
The shim answers `image inspect sha256:X` from RepoDigests and rewrites
|
||||
bare `sha256:X` run refs to `repo@sha256:X`.
|
||||
2. **LogConfig law** — podman inspect answers
|
||||
`HostConfig.LogConfig.Config: null` where docker returns the log-opts
|
||||
map (the doctor crashes on `.get("Config", {}) == null`). The shim
|
||||
records `run --log-opt` pairs and normalizes inspect output.
|
||||
3. **docker info law** — podman lacks `SecurityOptions`/`ServerVersion`;
|
||||
the shim answers rootless + `27.5.1` (liveness probe).
|
||||
- venv: `~/.r9v-venv` (huggingface_hub for fetch/verify)
|
||||
|
||||
## Bring-up battery (qualification, 2026-09-27)
|
||||
|
||||
- Setup `ready: true`; all 20 payload artifacts size+sha256 verified
|
||||
(175.30 GiB total across the six shards).
|
||||
- Vision: 64×64 red PNG → "Red" (79 tok, 7.4 s end-to-end on the scratch
|
||||
port). Prose: 100 tok in 9.1 s ≈ 11.0 tok/s wall **on a cold cache with
|
||||
reasoning tokens included in the wall clock** — decode-rate context, not
|
||||
a bench (no throughput bench was run, per Frank's direction).
|
||||
- MTP4: mean acceptance length 3.19–5.00 across the battery.
|
||||
- PLE pinned proof: `mapping_bytes = mapping_rss_bytes =
|
||||
54,400,261,120` (fully RSS-resident); doubling for re-pin would need
|
||||
108,800,522,240 ≤ available 140,876,808,192; registration took 446 s.
|
||||
|
||||
## Memory shape (measured)
|
||||
|
||||
- Load: 21.84 GiB / worker at ~158 s; cold start → serve ~16 min.
|
||||
- PLE pinned residency 54.4 GB in host RAM (vs the uncensored payload's
|
||||
ssd-mmap residency).
|
||||
- Context envelope: `R9V_MAX_MODEL_LEN=131072`; harness `models.yml`
|
||||
carries `contextWindow: 131072`, `maxTokens: 36864` (compaction-gap
|
||||
math: 70% × 131,072 = 91,750 + 36,864 = 128,614 ≤ 131,072).
|
||||
- Prefix caching OFF (`R9V_ENABLE_PREFIX_CACHING=0`), CACHE192 off — the
|
||||
KV envelope at 131,072 is smaller than the uncensored 196,608 shape.
|
||||
|
||||
## Ops
|
||||
|
||||
- **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve
|
||||
script re-gates everything, relaunch through the shim, waits).
|
||||
- **First start after boot** re-warms ~175 GiB of page cache; expect
|
||||
minutes to /health; `TimeoutStartSec=1800`.
|
||||
- **Scratch/probe**: NEVER probe against the prod state dir. Copy it, set
|
||||
`R9V_HOST_PORT` to a free port and `R9V_CONTAINER_NAME` to a scratch
|
||||
name, and stop the probe before any deploy — a probe holding :7770
|
||||
produces a false-green deploy (the 2026-09-26 lesson, guarded by the
|
||||
MainPID check).
|
||||
- **Rollback to uncensored**: point
|
||||
`scripts/frank-primary.env` `R9V_STATE_FILE` back at
|
||||
`/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys
|
||||
(adapter, FULL_MUTABLE_CACHE) and the template keys, redeploy — the
|
||||
uncensored state dir and payload are intact on disk.
|
||||
- **Models on the wire**: served alias `frank-primary`; thinking channel
|
||||
is `reasoning` (vLLM); tool parser `qwen3_coder`.
|
||||
|
||||
## Known costs (accepted at cutover)
|
||||
|
||||
- **Residency-halved experts** — Q8_0 doubles expert bytes vs IQ4_XS, so
|
||||
the tiered cache holds half as many experts in VRAM at the same budget;
|
||||
the probe's 11 tok/s decode was residency-bound, and the production
|
||||
number depends on the re-qualified budgets in the q8 state.
|
||||
- **128K envelope** — down from the uncensored 196,608 (which itself was
|
||||
down from 262,144 on llama.cpp).
|
||||
- **Reference quant** — back to the unsloth Q8_0 weights (no
|
||||
abliteration); answer-quality profile differs from the uncensored
|
||||
incumbent by design.
|
||||
- **Docker-pinned fork** — updates flow through image bundles + the shim,
|
||||
never a bare container run; the launcher stays the single argv owner.
|
||||
@@ -3,13 +3,19 @@ title: frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS
|
||||
service: frank-primary
|
||||
host: dankserver
|
||||
port: 7770
|
||||
status: active
|
||||
deployed: 2026-09-27
|
||||
status: retired
|
||||
retired: 2026-09-27
|
||||
superseded_by: "[[frank-primary-r9v-radiance-qwen38-q8]]"
|
||||
supersedes: "[[frank-primary-llamacpp-qwen38-flash-next-q8]]"
|
||||
tags: [llm, frank-primary, r9v, vllm, radiance, gfx1201]
|
||||
tags: [llm, frank-primary, r9v, vllm, radiance, gfx1201, retired]
|
||||
---
|
||||
|
||||
# frank-primary — R9V radiance-vLLM (dankserver, ACTIVE)
|
||||
# frank-primary — R9V radiance-vLLM (dankserver, RETIRED)
|
||||
|
||||
> **RETIRED 2026-09-27** — superseded the same day by the Q8_0 cutover,
|
||||
> [[frank-primary-r9v-radiance-qwen38-q8]]. Kept as the rollback runbook:
|
||||
> the uncensored state dir, frozen config and payload are intact on disk;
|
||||
> the rollback procedure lives in that page's Ops section.
|
||||
|
||||
The Dankserver LLM endpoint is the **R9V radiance-vLLM fork**
|
||||
(`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt) serving
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-09-27):
|
||||
**Dankserver runs R9V radiance-vLLM on frank-primary** (the drwolfen fork
|
||||
serving the abliterated Dyluhn Qwen3.8-Flash-Next-Uncensored IQ4_XS + MTP4
|
||||
across both Radeon AI PRO R9700s on :7770, docker-pinned runtime driven
|
||||
through a docker→podman shim), and Dankclaw runs two llama.cpp endpoints —
|
||||
serving the unsloth Qwen3.8-Flash-Next Q8_0 + MTP4 across both Radeon AI
|
||||
PRO R9700s on :7770, docker-pinned runtime driven through a docker→podman
|
||||
shim, the Qwen-Sharp template, 131072 ctx), and Dankclaw runs two llama.cpp endpoints —
|
||||
stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth
|
||||
Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151
|
||||
iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B
|
||||
@@ -25,8 +25,8 @@ gfx1201, 32 GiB each]
|
||||
B2[R9V radiance-vLLM fork, docker-pinned
|
||||
via docker-podman shim
|
||||
frank-primary, port 7770]
|
||||
C[Qwen3.8-Flash-Next-Uncensored IQ4_XS
|
||||
TP=2 tiered experts, MTP4, CED, 196608 ctx, vision]
|
||||
C[Qwen3.8-Flash-Next Q8_0
|
||||
TP=2 tiered experts, MTP4, sharp template, 131072 ctx, vision]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
@@ -58,16 +58,27 @@ frank-small, port 7771]
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-primary-r9v-radiance-qwen38-uncensored]] — the ACTIVE
|
||||
frank-primary runbook (2026-09-27, root systemd unit, port 7770): the
|
||||
R9V radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned
|
||||
- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
|
||||
runbook (2026-09-27, root systemd unit, port 7770): the R9V
|
||||
radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned
|
||||
image bundle `2dac17a2`, driven through a docker→podman shim) serving
|
||||
the unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) +
|
||||
blockfp8 MTP4 (depth 4) + vision on BOTH R9700s — TP=2 tiered experts
|
||||
(80-slot LRU cache), PLE n-gram PINNED in host RAM (54.4 GB), CED off,
|
||||
131072 ctx, the Qwen-Sharp v22.4.0 template via the launcher's
|
||||
`R9V_CHAT_TEMPLATE_FILE` passthrough (fork `30d4a32`), the quantized
|
||||
inject-shard dev-overlay adapter (fork `363d606`), reasoning channel
|
||||
`reasoning`, parser `qwen3_coder`. The tuned shape lives in the
|
||||
on-host q8xl setup state. Rollback:
|
||||
[[frank-primary-r9v-radiance-qwen38-uncensored]]
|
||||
- [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
|
||||
(same-day swap to the Q8_0 payload; rollback target, state + payload
|
||||
intact on disk): the R9V radiance-vLLM fork serving
|
||||
Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
|
||||
blockfp8 MTP4 (depth 4) + F16 vision on BOTH R9700s — TP=2 tiered
|
||||
experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428
|
||||
static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx, reasoning
|
||||
channel `reasoning`, parser `qwen3_coder`. Root unit + LimitMEMLOCK;
|
||||
the tuned 104-key shape lives in the on-host setup state. Rollback:
|
||||
static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx. Cut over
|
||||
from llama.cpp the same day; rollback target
|
||||
[[frank-primary-llamacpp-qwen38-flash-next-q8]]
|
||||
- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
|
||||
(rollback target; the llama.cpp b11030 endpoint of 2026-09-26):
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# Bundle Change Log
|
||||
|
||||
## 2026-09-27
|
||||
- `/scripts/frank-primary.env`, `/scripts/frank-primary.service`, `/scripts/serve-frank-primary.sh`, `/scripts/deploy-remote.sh`, `/tests/frank_primary_unit.bats`, `/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` (new), AGENTS.md + LLM index — **frank-primary cut over to the Q8_0 payload** (uncensored IQ4_XS → unsloth Qwen3.8-Flash-Next Q8_0, 175.30 GiB / 6 shards) at Frank's direction. Launcher gained the opt-in `R9V_CHAT_TEMPLATE_FILE`/`R9V_CHAT_TEMPLATE_KWARGS` passthrough (fork commit `30d4a32`, local-only) — the Qwen-Sharp v22.4.0 template now serves, fleet parity with frank-large. Env points at `/home/dank/r9v-state/q8xl/setup.json` + the quantized-inject dev-overlay adapter (fork `363d606`) + `R9V_FULL_MUTABLE_CACHE=0`; shard gates extended to SHARD4/5/6 (unset-safe); seams recalibrated; ctx window 131072 (models.yml maxTokens 36864). Spec decode is MTP4 only — the fork's `speculative_config` is a single dict, no chaining, so llama.cpp's `draft-mtp,ngram-mod` pattern is NOT expressible (PLE n-gram rides along architecturally). Uncensored state + payload remain on disk as the rollback target. 346 tests green.
|
||||
- Dankserver frank-primary slot (experiment window) — the `radiance-vllm-qwen4exp` fork qualified the unsloth `Qwen3.8-Flash-Next-GGUF` **Q8_0** payload (175.30 GiB, 6 split parts) on the dual R9700. Two launch blockers fixed loader-side in the fork: `runtimes/dev-overrides/qwen4_exp.py` dequantizes quantized `block_inject` shards in-stream (the Q8_0 payload quantizes BOTH fused-HC shards; IQ4_XS ships inject as F32 — the uniformity check rejected the mixed projection) and `R9V_FULL_MUTABLE_CACHE=0` passthrough for the 80-slot tiered cache (fork commits 363d606, 2fe6dbb, 50a79ec — GitHub remote, not pushed). Battery on :7773: text + reasoning channel, vision ("Red" on a red PNG), prose ≈11.0 tok/s wall, MTP4 mean acceptance length 3.19–5.00; PLE pinned residency proven (`mapping_bytes=54400261120`, `mapping_rss_bytes` == mapping size, both ranks required 108.8 GiB vs 140.9 available). Relaunch recipe saved at `dankserver:/home/dank/r9v-state/q8xl/START-COMMAND.txt`; record at fork `docs/qualification/qwen38-q8-0-dual-r9700-2026-09-27.md`.
|
||||
- Dankserver frank-primary restore — uncensored UD-IQ4_XS back on `0.0.0.0:7770` as `frank-primary-r9v` from its frozen config (health 200 in ~60 s; LAN-path completion returns content + reasoning); `frank-primary.service` (llama.cpp) **runtime-masked** for the window (`/run/systemd/system/frank-primary.service → /dev/null` — the `/etc` relink symlink blocks a plain mask; self-clears at reboot, unit file untouched) and the container runs `restart=no`, so a reboot returns the repo-configured llama.cpp with no contention. Restore config preserved at `dankserver:/home/dank/r9v-state/uncensored/restore-config-20260927.env`; scratch in `dankserver:~/.AGENTTRASH/20260927-q8xl-bringup/`.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user