docs(frank-large): unsloth Qwen3.8-Flash-Next runbook; retire the CIRU Orca runbook
New runbook frank-large-unsloth-qwen38-flash-next.md (runtime, payload, deployed config, the explicit -md, the 114 GiB GTT memory model, client contract, rollback). frank-large-ciru-orca.md marked RETIRED (rollback target, payload and runtime intact on disk). AGENTS.md topology row, frank-large section, harness note, runbook index, and the GPU-budget lesson (GTT aperture ~114 GiB, not the stale 64 GiB rocm-smi figure — the CIRU Orca held 92.6 GiB on the same device). LLM index header, mermaid, active list, and the vault root index updated. Work-log entry.
This commit is contained in:
@@ -12,7 +12,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service.
|
||||
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
|
||||
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (CIRU v4.4.1 llama.cpp on
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (unsloth llama.cpp prebuilt on
|
||||
the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
|
||||
Dankclaw reverse-proxies both hosts.
|
||||
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
|
||||
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | orcarouter Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards) with the z-lab DFlash2 external drafter (depth 7, live mean accept length 4.24/7 — A/B tested and beats MTP self-speculation on decode: 100.0 vs 85.9 t/s), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 384,238-token KV pool = 1.47x the window), fp8 KV, prefix caching, vision | 7770 |
|
||||
| Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 |
|
||||
| Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-Flash-Next UD-IQ4_XS on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 200000 ctx single stream (1 slot), MTP depth 2 via the explicit `-md` shared-Q8_0 head, text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
@@ -265,7 +265,7 @@ a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-09-20 both `frank-small` and `frank-medium` accept images
|
||||
(`input: [text, image]`) — frank-medium's checkpoint serves vision through the
|
||||
Gemma4 merge path with no separate mmproj, verified on held-out synthetic
|
||||
images; `frank-large` remains text-only because its CIRU deployment loads no
|
||||
images; `frank-large` remains text-only because its unsloth deployment loads no
|
||||
vision projector (the F16 mmproj is on disk, one env flip away). Note also that
|
||||
the reasoning channel differs by runtime: llama.cpp endpoints emit
|
||||
`reasoning_content`, vLLM emits `reasoning`.
|
||||
@@ -273,40 +273,45 @@ There is **no audio**
|
||||
input anywhere in the fleet: that capability left with the FLM NPU endpoint on
|
||||
2026-09-11.
|
||||
|
||||
### frank-large — CIRU v4.4.1 Orca (Dankclaw, ACTIVE)
|
||||
### frank-large — unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS (Dankclaw, ACTIVE)
|
||||
|
||||
CIRU v4.4.1 (the QSA window hotfix) as a **root systemd unit** (`scripts/
|
||||
frank-large.service`, `User=dank`), serving `jcbtc/Qwen3.8-Flash-CIRU-STRIX-
|
||||
Orca` (79.4 GiB main GGUF + a 4.1 GiB MTP head + a 52.4 GiB PLE payload, all
|
||||
sha256-verified) on port 7770 from the gfx1151 iGPU. One stream, the model's
|
||||
FULL 262,144 trained window (`PARALLEL_SLOTS=1`; the MTP drafter needs exactly
|
||||
one slot in this runtime), MTP depth 4, PLE cache 4096 MiB, reasoning extracted
|
||||
to `reasoning_content`, and **no vision projector** (the F16 mmproj is on disk,
|
||||
`ENABLE_VISION=0` — one env flip to enable). It replaced halogen-flash-server,
|
||||
which had replaced ik_llama.cpp.
|
||||
The unsloth llama.cpp prebuilt `app-b11030-mix-5ff778e` (ROCm gfx1151,
|
||||
`/home/dank/GitPackages/unsloth-llama-b11030/`, bundles its own ROCm runtime —
|
||||
no host SDK gate, no venv) as a **root systemd unit** (`scripts/
|
||||
frank-large.service`, `User=dank`), serving
|
||||
`unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (87.3 GiB main GGUF, three HF
|
||||
split parts concatenated, + a 2.6 GiB MTP head, all sha256-verified) on port
|
||||
7770 from the gfx1151 iGPU. One stream, 200,000 ctx (`PARALLEL_SLOTS=1`,
|
||||
operator's requested shape; the trained window is 262,144), MTP depth 2
|
||||
(`SPEC_DRAFT_N_MAX=2`, the model card's default), reasoning extracted to
|
||||
`reasoning_content`, and **no vision projector** (the F16 mmproj is on disk,
|
||||
`ENABLE_VISION=0` — one env flip to enable). It replaced the CIRU v4.4.1
|
||||
Orca runtime, which had replaced halogen-flash-server.
|
||||
|
||||
The runtime is a **private host build** (`/home/dank/GitPackages/ciru-runtime-
|
||||
v4.4.1-qsa-hotfix/`), not a container image: a gfx1151 HIP build of the CIRU
|
||||
llama.cpp fork with its own bundled ROCm SDK in a venv (`.venv-rocm/`), and the
|
||||
binary resolves the SDK libs via RUNPATH. The launcher (`scripts/ciru/
|
||||
run-orca-server.sh` → `run-server.sh`) has a **runtime gate**: it exits 2 unless
|
||||
`${repo_root}/runtime/hip/lib/libamdhip64.so` and `runtime/rocr/lib/
|
||||
libhsa-runtime64.so` exist. The serve script (`scripts/serve-frank-large.sh`)
|
||||
satisfies that gate idempotently by `ln -sfn`-ing both to the venv's unversioned
|
||||
devel symlinks, then `exec`s the launcher with `--chat-template-kwargs
|
||||
'{"preserve_thinking":true}'` (Qwen3.8's template needs this to carry thinking
|
||||
blocks across turns; a request-level value still wins).
|
||||
The runtime is the first with an MTP graph for the qwen4exp architecture
|
||||
(stock ggml-org mainline has none), so the unsloth prebuilt is the
|
||||
llama.cpp-compatible choice. The MTP head is loaded via an **explicit
|
||||
`-md`**: sidecar auto-discovery does not search the `mtp/` subfolder, and
|
||||
without it `--spec-type draft-mtp` silently runs at the main model's speed.
|
||||
The serve script (`scripts/serve-frank-large.sh`) validates binary → main
|
||||
GGUF → MTP head and exits with a pointer message if any is missing, then
|
||||
`exec`s `llama-server` with `--jinja --alias frank-large --chat-template-
|
||||
kwargs '{"preserve_thinking":true}'` (Qwen3.8's template needs this to
|
||||
carry thinking blocks across turns; a request-level value still wins).
|
||||
|
||||
The serve script validates launcher → built binary → venv runtime → main GGUF →
|
||||
MTP head → PLE payload + manifest + scale, and exits with a pointer message if
|
||||
any is missing. `PLE_CACHE_MIB=4096` is the vendor production-profile default;
|
||||
the 52.4 GiB PLE is mmap'd and paged, not resident.
|
||||
The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
|
||||
`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated"
|
||||
is not the budget). The 87.3 GiB model is mmap'd in with `-ngl 999`; budget
|
||||
at 200k ≈ 100 GiB (weights + MTP head + KV + buffers), leaving ~14 GiB of
|
||||
aperture headroom. The n-gram embedding (PLE) is inside the GGUF — no
|
||||
external PLE payload or pager, unlike the CIRU Orca packaging.
|
||||
|
||||
The halogen Quadlet (`dankclaw/frank-large/quadlet/`) was retired to
|
||||
`.AGENTTRASH/` — one owner per endpoint, now the scripts/ trio.
|
||||
Fetch: `scripts/download-frank-large-unsloth.sh` (resumable, each part
|
||||
verified against its LFS sha256 fetched live from the Hub tree API, then the
|
||||
assembled GGUF against `SHA256SUMS`; `--verify-only` re-checks).
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md`.
|
||||
Rollback: [[halogen-flash-server]].
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md`.
|
||||
Rollback: [[frank-large-ciru-orca]] (payload and runtime intact on disk).
|
||||
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
@@ -530,9 +535,13 @@ linked runbooks.
|
||||
- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on
|
||||
qwen4exp needs `scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on
|
||||
the agentionai FAST layer set; the kingjones STRIX set works unpatched.
|
||||
- **GPU budget on Dankclaw is the rocm-smi figure** (64 GiB, SVM-backed).
|
||||
`free`/`dmidecode` show the CPU-side carve-out — intentional, do not
|
||||
change.
|
||||
- **GPU budget on Dankclaw is the GTT aperture (~114 GiB), not the
|
||||
rocm-smi "dedicated" figure.** The Radeon 8060S (gfx1151) reports
|
||||
116,732 MiB (`llama-server --list-devices`;
|
||||
`gtt_total=122402963456` in `/sys/class/drm/card1/device/`). The older
|
||||
"64 GiB, SVM-backed" lesson was already stale — the CIRU Orca predecessor
|
||||
held 92.6 GiB of weights on the iGPU. `free`/`dmidecode` show the
|
||||
CPU-side carve-out — intentional, do not change.
|
||||
|
||||
**Load / crash diagnosis**
|
||||
|
||||
@@ -743,7 +752,7 @@ Active:
|
||||
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (CIRU v4.4.1 Orca, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` |
|
||||
| frank-large (unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md` |
|
||||
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
|
||||
| frank-medium (radiance TP=2 + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
|
||||
| frank-medium model swap (toolchain + process) | `serverAppConfigs-docs/LLM/frank-medium-model-swap.md` |
|
||||
@@ -757,6 +766,7 @@ Retired (rollback targets and history):
|
||||
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
|
||||
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
|
||||
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
|
||||
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
|
||||
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
|
||||
| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
|
||||
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
||||
|
||||
Reference in New Issue
Block a user