dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: frank-primary q8 cutover — new runbook, uncensored retired as rollback, fleet table/index/log synced

Browse Source
This commit is contained in:
Frank committed 2026-09-27 22:53:19 -07:00
1 parent ca2810a319
commit 671f784ca4
5 files changed
+260 -49
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+50 -34
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 GGUF shards) with the blockfp8 MTP4 head (depth 4) and F16 mmproj, TP=2 tiered experts (UVA pinned host copy 43 GiB rank 0 + 16 GiB PLE pinned reserve; PLE n-gram `ssd` mmap; budgets 146.25/78.75 GiB by device), CED on, 196608 ctx single stream, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 |
| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 |
| Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), q8_0 KV, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
@@ -345,29 +345,40 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
Rollback: [[frank-small-qwen38-gsq-rco]].
### frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (Dankserver, ACTIVE)
### frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0 (Dankserver, ACTIVE)
The **R9V radiance-vLLM fork** (`drwolfen/radiance-vllm-qwen4exp`, docker-pinned
prebuilt image bundle `r9v-qwen38-flash-next-wmma@sha256:2dac17a2…`) serving
`Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS` (abliterated weights,
92.39 GiB, 3 GGUF shards) with the blockfp8 MTP4 head (depth 4) and F16
mmproj on **both** Radeon AI PRO R9700s (gfx1201, BDFs `0000:03:00.0` /
`0000:07:00.0`), one stream at **196,608 ctx**, **TP=2 tiered experts**
(UVA pinned host copy ~43 GiB on rank 0 + 16 GiB PLE pinned reserve; PLE
n-gram table on `ssd` mmap residency; per-device offload budgets
146.25/78.75 GiB), **CED on** (split-16 projector engagement at >=8192-token
prompts), tool parser `qwen3_coder`, reasoning channel `reasoning`. Cut
over 2026-09-27 from the llama.cpp b11030 endpoint (downtime ~1 h 24 m) at
Frank's direction.
the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** (175.30 GiB, 6 GGUF shards)
with the blockfp8 MTP4 head (depth 4) and the mmproj on **both** Radeon AI
PRO R9700s (gfx1201, BDFs `0000:03:00.0` / `0000:07:00.0`), one stream at
**131,072 ctx**, **TP=2 tiered experts** (80-slot LRU expert cache; PLE
n-gram table PINNED in host RAM, 54.4 GB), the **Qwen-Sharp v22.4.0
template** (via the launcher's `R9V_CHAT_TEMPLATE_FILE` passthrough, fork
`30d4a32`), tool parser `qwen3_coder`, reasoning channel `reasoning`. Cut
over 2026-09-27 from the uncensored IQ4_XS payload (itself cut over from
llama.cpp b11030 the same day) at Frank's direction.
**The tuned shape lives on-host, not in the repo.** The profile's setup run
qualified a 104-key `R9V_*` config (placement, budgets, parsers, CED) into
`/home/dank/r9v-state/uncensored/setup.json` (`ready: true`); the repo env
**The tuned shape lives on-host, not in the repo.** The q8 profile's setup
run qualified the `R9V_*` config into
`/home/dank/r9v-state/q8xl/setup.json` (`ready: true`); the repo env
file `scripts/frank-primary.env` carries ONLY production overrides
(`R9V_HOST_BIND=0.0.0.0`, `R9V_HOST_PORT=7770`,
`R9V_SERVED_MODEL_NAME=frank-primary`, `R9V_CONTAINER_NAME=frank-primary-r9v`,
`R9V_EXPECTED_GPU_BDFS`). Never copy tuned values from the state into the
repo — it would drift from the qualification.
`R9V_EXPECTED_GPU_BDFS`) plus two q8-only keys: the dev-overlay adapter
(`R9V_DEV_QWEN4_EXP_ADAPTER_PY`, fork commit `363d606` — the Q8_0 payload
quantizes BOTH fused hyper-connection shards, so the shard-uniformity
check needs the in-stream dequant) and `R9V_FULL_MUTABLE_CACHE=0` (the
overlay's env.always injects the uncensored 160-slot cache geometry, which
the q8 80-slot tiered cache rejects). Never copy tuned values from the
state into the repo — it would drift from the qualification.
**Spec decoding is MTP4 only.** The fork's `speculative_config` is a
single dict — one method per engine, NO chaining — so the llama.cpp
`draft-mtp,ngram-mod` pattern is NOT expressible here. MTP4 is wired
(`R9V_MTP_SPEC_TOKENS=4`, draft TP=2); the PLE n-gram rides along
architecturally (`ngram_embedding`); `ngram_gpu` exists as an alternative
single method, not a chainable one.
**Serving = the fork's own launcher through a docker→podman shim.** The R9V
tooling is docker-pinned; the shim at
@@ -383,9 +394,9 @@ and normalizes inspect output; (3) podman lacks docker's `SecurityOptions`
/`ServerVersion` info fields — the shim answers them.
**Unit and self-healing.** Root systemd unit (`scripts/frank-primary.service`,
`User=dank`, `LimitMEMLOCK=infinity`) — the tiered-expert UVA host copy pins
~43 GiB and the PLE reserve 16 GiB, and a root unit grants the pin budget
via PID1 (a rootless quadlet would be capped by the user manager's limit).
`User=dank`, `LimitMEMLOCK=infinity`) — the PLE pinned residency alone is
54.4 GB, and a root unit grants the pin budget via PID1 (a rootless
quadlet would be capped by the user manager's limit).
The serve script (`scripts/serve-frank-primary.sh`) gates the setup state
(ready + paths), the pinned image (RepoDigests), the PLE byte count
(`stat -c %s`), every GGUF shard + the mmproj and the GPU device nodes,
@@ -398,19 +409,22 @@ PLE/shards, `podman rm -f`s the container, waits out :7770, relinks/enables/
restarts the unit and health-gates with the MainPID check (the false-green
stray law).
Verified at cutover: setup PASS=28/WARN=3/FAIL=0 (20/20 artifacts
sha256-verified, PLE at pinned size); first-start doctor battery PASS=34/
WARN=8/FAIL=0 including pinned-UVA reads, H2D copies and the TP all-reduce
verified INSIDE serving workers; wire: alias `frank-primary`, `/health` +
`/metrics` 200, non-thinking PONG, thinking → `reasoning`, tool_calls
round-trip, vision ('red' for a red PNG), needle recall exact at 111,647
prompt tokens (95% depth, ~4.5k tok/s effective prefill), decode 59–96
tok/s (draft-acceptance dependent). MemAvailable floor 56.3 GiB.
Verified at cutover (2026-09-27): q8 setup state `ready: true`, all 20
payload artifacts sha256-verified (175.30 GiB); the scratch-port bring-up
battery answered vision ("Red" for a 64×64 red PNG), MTP4 mean acceptance
length 3.19–5.00, and the PLE pinned proof (`mapping_bytes =
mapping_rss_bytes = 54,400,261,120`, fully RSS-resident); wire + tool
round-trip + template probe re-verified on :7770 after the deploy. No
throughput bench was run (Frank declined); the probe-era 11 tok/s decode
was residency-bound (halved expert headroom at double the bytes/expert),
not the production number.
Runbook: `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-uncensored.md`.
Rollback: the llama.cpp trio at commit `4992e5a` (unit/env/serve) and the
GGUF payload on disk — see [[frank-primary-llamacpp-qwen38-flash-next-q8]];
a rollback is a redeploy.
Runbook: `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md`.
Rollback: the uncensored payload — flip `R9V_STATE_FILE` back to
`/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys
and the template keys, redeploy (state + `restore-config-20260927.env` +
payload intact on disk) — see
[[frank-primary-r9v-radiance-qwen38-uncensored]].
## Key Lessons Learned
Durable operational lessons from the fleet's history. Full detail in the
@@ -759,7 +773,7 @@ Active:
|---------|---------|
| frank-large (stock llama.cpp Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md` |
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
| frank-primary (llama.cpp Qwen3.8-Flash-Next Q8_0 + shared MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md` |
| frank-primary (R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
@@ -767,10 +781,12 @@ Retired (rollback targets and history):
| Runtime | Host | Window | Runbook |
|---------|------|--------|---------|
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
| llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` |
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
| vllm-radiance TP=2 MXFP4 + DFlash2 (`frank-medium`) | Dankserver | 2026-09-20 → 09-26 | `frank-medium-radiance-mxfp4.md` |
| Lucebox DS4.1 DeepSeek-V4.1-Flash ROCmFP23 (`frank-primary`) | Dankserver | 2026-09-26 (same-day swap) | `frank-primary-lucebox-ds4.md` |
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
| unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` |