docs(frank-large): unsloth Qwen3.8-Flash-Next runbook; retire the CIRU Orca runbook

New runbook frank-large-unsloth-qwen38-flash-next.md (runtime, payload,
deployed config, the explicit -md, the 114 GiB GTT memory model, client
contract, rollback). frank-large-ciru-orca.md marked RETIRED (rollback
target, payload and runtime intact on disk). AGENTS.md topology row,
frank-large section, harness note, runbook index, and the GPU-budget
lesson (GTT aperture ~114 GiB, not the stale 64 GiB rocm-smi figure —
the CIRU Orca held 92.6 GiB on the same device). LLM index header,
mermaid, active list, and the vault root index updated. Work-log entry.
This commit is contained in:
Frank
2026-09-21 12:30:35 -07:00
parent b149d02187
commit 7010f5b754
6 changed files with 307 additions and 63 deletions
+46 -36
View File
@@ -12,7 +12,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service.
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (CIRU v4.4.1 llama.cpp on
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (unsloth llama.cpp prebuilt on
the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
Dankclaw reverse-proxies both hosts.
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | orcarouter Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards) with the z-lab DFlash2 external drafter (depth 7, live mean accept length 4.24/7 — A/B tested and beats MTP self-speculation on decode: 100.0 vs 85.9 t/s), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 384,238-token KV pool = 1.47x the window), fp8 KV, prefix caching, vision | 7770 |
| Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 |
| Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-Flash-Next UD-IQ4_XS on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 200000 ctx single stream (1 slot), MTP depth 2 via the explicit `-md` shared-Q8_0 head, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
Retired endpoints (`frank-npu`, and every prior runtime) are in
@@ -265,7 +265,7 @@ a role pointing at a provider the file does not define fails at call time. As
of 2026-09-20 both `frank-small` and `frank-medium` accept images
(`input: [text, image]`) — frank-medium's checkpoint serves vision through the
Gemma4 merge path with no separate mmproj, verified on held-out synthetic
images; `frank-large` remains text-only because its CIRU deployment loads no
images; `frank-large` remains text-only because its unsloth deployment loads no
vision projector (the F16 mmproj is on disk, one env flip away). Note also that
the reasoning channel differs by runtime: llama.cpp endpoints emit
`reasoning_content`, vLLM emits `reasoning`.
@@ -273,40 +273,45 @@ There is **no audio**
input anywhere in the fleet: that capability left with the FLM NPU endpoint on
2026-09-11.
### frank-large — CIRU v4.4.1 Orca (Dankclaw, ACTIVE)
### frank-large — unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS (Dankclaw, ACTIVE)
CIRU v4.4.1 (the QSA window hotfix) as a **root systemd unit** (`scripts/
frank-large.service`, `User=dank`), serving `jcbtc/Qwen3.8-Flash-CIRU-STRIX-
Orca` (79.4 GiB main GGUF + a 4.1 GiB MTP head + a 52.4 GiB PLE payload, all
sha256-verified) on port 7770 from the gfx1151 iGPU. One stream, the model's
FULL 262,144 trained window (`PARALLEL_SLOTS=1`; the MTP drafter needs exactly
one slot in this runtime), MTP depth 4, PLE cache 4096 MiB, reasoning extracted
to `reasoning_content`, and **no vision projector** (the F16 mmproj is on disk,
`ENABLE_VISION=0` — one env flip to enable). It replaced halogen-flash-server,
which had replaced ik_llama.cpp.
The unsloth llama.cpp prebuilt `app-b11030-mix-5ff778e` (ROCm gfx1151,
`/home/dank/GitPackages/unsloth-llama-b11030/`, bundles its own ROCm runtime —
no host SDK gate, no venv) as a **root systemd unit** (`scripts/
frank-large.service`, `User=dank`), serving
`unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (87.3 GiB main GGUF, three HF
split parts concatenated, + a 2.6 GiB MTP head, all sha256-verified) on port
7770 from the gfx1151 iGPU. One stream, 200,000 ctx (`PARALLEL_SLOTS=1`,
operator's requested shape; the trained window is 262,144), MTP depth 2
(`SPEC_DRAFT_N_MAX=2`, the model card's default), reasoning extracted to
`reasoning_content`, and **no vision projector** (the F16 mmproj is on disk,
`ENABLE_VISION=0` — one env flip to enable). It replaced the CIRU v4.4.1
Orca runtime, which had replaced halogen-flash-server.
The runtime is a **private host build** (`/home/dank/GitPackages/ciru-runtime-
v4.4.1-qsa-hotfix/`), not a container image: a gfx1151 HIP build of the CIRU
llama.cpp fork with its own bundled ROCm SDK in a venv (`.venv-rocm/`), and the
binary resolves the SDK libs via RUNPATH. The launcher (`scripts/ciru/
run-orca-server.sh` → `run-server.sh`) has a **runtime gate**: it exits 2 unless
`${repo_root}/runtime/hip/lib/libamdhip64.so` and `runtime/rocr/lib/
libhsa-runtime64.so` exist. The serve script (`scripts/serve-frank-large.sh`)
satisfies that gate idempotently by `ln -sfn`-ing both to the venv's unversioned
devel symlinks, then `exec`s the launcher with `--chat-template-kwargs
'{"preserve_thinking":true}'` (Qwen3.8's template needs this to carry thinking
blocks across turns; a request-level value still wins).
The runtime is the first with an MTP graph for the qwen4exp architecture
(stock ggml-org mainline has none), so the unsloth prebuilt is the
llama.cpp-compatible choice. The MTP head is loaded via an **explicit
`-md`**: sidecar auto-discovery does not search the `mtp/` subfolder, and
without it `--spec-type draft-mtp` silently runs at the main model's speed.
The serve script (`scripts/serve-frank-large.sh`) validates binary → main
GGUF → MTP head and exits with a pointer message if any is missing, then
`exec`s `llama-server` with `--jinja --alias frank-large --chat-template-
kwargs '{"preserve_thinking":true}'` (Qwen3.8's template needs this to
carry thinking blocks across turns; a request-level value still wins).
The serve script validates launcher → built binary → venv runtime → main GGUF →
MTP head → PLE payload + manifest + scale, and exits with a pointer message if
any is missing. `PLE_CACHE_MIB=4096` is the vendor production-profile default;
the 52.4 GiB PLE is mmap'd and paged, not resident.
The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated"
is not the budget). The 87.3 GiB model is mmap'd in with `-ngl 999`; budget
at 200k ≈ 100 GiB (weights + MTP head + KV + buffers), leaving ~14 GiB of
aperture headroom. The n-gram embedding (PLE) is inside the GGUF — no
external PLE payload or pager, unlike the CIRU Orca packaging.
The halogen Quadlet (`dankclaw/frank-large/quadlet/`) was retired to
`.AGENTTRASH/` — one owner per endpoint, now the scripts/ trio.
Fetch: `scripts/download-frank-large-unsloth.sh` (resumable, each part
verified against its LFS sha256 fetched live from the Hub tree API, then the
assembled GGUF against `SHA256SUMS`; `--verify-only` re-checks).
Runbook: `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md`.
Rollback: [[halogen-flash-server]].
Runbook: `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md`.
Rollback: [[frank-large-ciru-orca]] (payload and runtime intact on disk).
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
@@ -530,9 +535,13 @@ linked runbooks.
- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on
qwen4exp needs `scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on
the agentionai FAST layer set; the kingjones STRIX set works unpatched.
- **GPU budget on Dankclaw is the rocm-smi figure** (64 GiB, SVM-backed).
`free`/`dmidecode` show the CPU-side carve-out — intentional, do not
change.
- **GPU budget on Dankclaw is the GTT aperture (~114 GiB), not the
rocm-smi "dedicated" figure.** The Radeon 8060S (gfx1151) reports
116,732 MiB (`llama-server --list-devices`;
`gtt_total=122402963456` in `/sys/class/drm/card1/device/`). The older
"64 GiB, SVM-backed" lesson was already stale — the CIRU Orca predecessor
held 92.6 GiB of weights on the iGPU. `free`/`dmidecode` show the
CPU-side carve-out — intentional, do not change.
**Load / crash diagnosis**
@@ -743,7 +752,7 @@ Active:
| Service | Runbook |
|---------|---------|
| frank-large (CIRU v4.4.1 Orca, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` |
| frank-large (unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md` |
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
| frank-medium (radiance TP=2 + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
| frank-medium model swap (toolchain + process) | `serverAppConfigs-docs/LLM/frank-medium-model-swap.md` |
@@ -757,6 +766,7 @@ Retired (rollback targets and history):
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |