docs: update frank-large to stock llama.cpp UD-Q8_K_XL + DFlash2

- AGENTS.md: fleet topology, intro, frank-large section, text-only note,
  benchmarking policy, runbook index (new runbook active; Kairic Edge
  retired as rollback target)
- homepage services.json: frank-large description -> UD-Q8_K_XL, 31.5 GiB,
  DFlash2 spec decode
- homepage services.json: ComfyUI url + showLink (complete prior edit)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
FrankandClaude committed 2026-09-21 21:53:53 -07:00
1 parent 7689092db8
commit 600812d459
2 files changed
+38 -39

No files matched your search

+35 -36
View File
@@ -12,7 +12,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service. Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service.
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (Kairic Edge runtime on Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on
the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
Dankclaw reverse-proxies both hosts. Dankclaw reverse-proxies both hosts.
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port | | Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------| |--------|---------|---------|-------|------|
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, 19.37 GiB) with the z-lab DFlash2 external drafter (depth 7), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations, UNQUANTIZED bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 from the FP8/fp8-KV shape to test whether fp8 KV had been degrading W4A8 output quality; auto-sized 486,464-token pool = 1.86x the window), prefix caching, vision | 7770 | | Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, 19.37 GiB) with the z-lab DFlash2 external drafter (depth 7), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations, UNQUANTIZED bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 from the FP8/fp8-KV shape to test whether fp8 KV had been degrading W4A8 output quality; auto-sized 486,464-token pool = 1.86x the window), prefix caching, vision | 7770 |
| Dankclaw | `frank-large` | Kairic Edge (ciru-ai/ROCmFPX v1.2, gfx1151, root systemd unit, built from source) | jcbtc Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense, 26 GiB total) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), MTP depth 4 + n-gram drafting (24/64/64), compatibility mode, text only | 7770 | | Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), q8_0 KV, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
Retired endpoints (`frank-npu`, and every prior runtime) are in Retired endpoints (`frank-npu`, and every prior runtime) are in
@@ -228,7 +228,7 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
merely wedged kernel usually still answers ARP. merely wedged kernel usually still answers ARP.
1. Stop and mask (self-healing would restart it): 1. Stop and mask (self-healing would restart it):
- Dankclaw `frank-large` (root unit, CIRU llama.cpp on the iGPU): - Dankclaw `frank-large` (root unit, stock llama.cpp on the iGPU):
`sudo systemctl stop frank-large && sudo systemctl mask frank-large` `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090): - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
`sudo systemctl stop frank-small && sudo systemctl mask frank-small` `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
@@ -265,38 +265,36 @@ a role pointing at a provider the file does not define fails at call time. As
of 2026-09-20 both `frank-small` and `frank-medium` accept images of 2026-09-20 both `frank-small` and `frank-medium` accept images
(`input: [text, image]`) — frank-medium's checkpoint serves vision through the (`input: [text, image]`) — frank-medium's checkpoint serves vision through the
Gemma4 merge path with no separate mmproj, verified on held-out synthetic Gemma4 merge path with no separate mmproj, verified on held-out synthetic
images; `frank-large` remains text-only because its Kairic Edge deployment loads no images; `frank-large` remains text-only because its stock llama.cpp deployment loads no
vision projector (the model has no mmproj). Note also that vision projector (the mmproj is not loaded). Note also that
the reasoning channel differs by runtime: llama.cpp endpoints emit the reasoning channel differs by runtime: llama.cpp endpoints emit
`reasoning_content`, vLLM emits `reasoning`. `reasoning_content`, vLLM emits `reasoning`.
There is **no audio** There is **no audio**
input anywhere in the fleet: that capability left with the FLM NPU endpoint on input anywhere in the fleet: that capability left with the FLM NPU endpoint on
2026-09-11. 2026-09-11.
### frank-large — Kairic Edge Qwen3.8-27B IU4 (W4A4) (Dankclaw, ACTIVE) ### frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE)
The Kairic Edge runtime (ciru-ai/ROCmFPX, branch The unsloth llama.cpp prebuilt `app-b11030-mix-5ff778e` (ROCm gfx1151,
`kairic-edge-qwen38-27b-v1.2`, commit `205a3e5`) as a **root systemd unit** `/home/dank/GitPackages/unsloth-llama-b11030/`, bundles its own ROCm
(`scripts/frank-large.service`, `User=dank`), built from source runtime — no host SDK gate) as a **root systemd unit**
(`scripts/build-kairic-edge-gfx1151.sh`, TheRock 7.15.0a20260718 toolchain, (`scripts/frank-large.service`, `User=dank`), serving
Composable Kernel `fdf4bb7f` + the gfx1151-iu4 patch), serving `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` (27B dense, 31.5 GiB,
`jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` (27B dense, IU4 W4A4, 15.5 GiB main sha256-verified) on port 7770 from the gfx1151 iGPU. One stream, full
GGUF + 7.99 GiB FFN sidecar + 1.88 GiB GDN sidecar + 0.70 GiB GDN-Output 262,144 ctx (the model's native window, `PARALLEL_SLOTS=1`), DFlash2 spec
sidecar, all sha256-verified) on port 7770 from the gfx1151 iGPU. One decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, depth 4,
stream, full 262,144 ctx (the model's native window, `PARALLEL_SLOTS=1`), `SPEC_DRAFT_N_MAX=4`), reasoning extracted to `reasoning_content`, and
MTP depth 4 (`SPEC_DRAFT_N_MAX=4`) + n-gram drafting (24/64/64), reasoning **no vision projector**. It replaced the Kairic Edge ROCmFPX build (jcbtc
extracted to `reasoning_content` (compatibility mode — the harness needs Qwen3.8-27B-IU4-Kairic-Edge), which had replaced the unsloth
sampling, tools and a reasoning budget, which fast-greedy rejects), and Qwen3.8-Flash-Next UD-IQ4_XS.
**no vision projector**. It replaced the unsloth llama.cpp prebuilt
(b11030, Qwen3.8-Flash-Next UD-IQ4_XS, 125B MoE), which had replaced the
CIRU v4.4.1 Orca runtime.
The spec method is native MTP depth 4 + n-gram drafting. DFlash is NOT The spec method is DFlash2 (block-diffusion drafter) at depth 4. The
available for this model (no `DRAFT_DFLASH` spec type in the Kairic Edge drafter is loaded via `--model-draft`; the chain is `--spec-type
runtime). The three PromptForge sidecars (FFN, GDN, GDN-Output) are wired draft-dflash` with `--spec-draft-device ROCm0`. KV is **q8_0** (the
via the `PROMPTFORGE_*` env block (the certified runner's exact env). The unsloth b11030 build does not support fp8 KV; q8_0 is the
serve script (`scripts/serve-frank-large.sh`) validates binary → main GGUF → highest-precision 8-bit option, ~8 GiB at 262k). The serve script
3 sidecars and exits with a pointer message if any is missing, then `exec`s (`scripts/serve-frank-large.sh`) validates binary → main GGUF → drafter
and exits with a pointer message if any is missing, then `exec`s
`llama-server` with `--jinja --alias frank-large --chat-template-kwargs `llama-server` with `--jinja --alias frank-large --chat-template-kwargs
'{"preserve_thinking":true}' --chat-template-file $TEMPLATE_FILE` (the '{"preserve_thinking":true}' --chat-template-file $TEMPLATE_FILE` (the
Qwen-Sharp v22.4.0 template; without `--chat-template-file` the binary Qwen-Sharp v22.4.0 template; without `--chat-template-file` the binary
@@ -305,16 +303,16 @@ falls back to the built-in `peg-native` template, which produces a
The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR, The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated" `gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated"
is not the budget). The 27B dense model (26 GiB total) is mmap'd in with is not the budget). The 27B dense model (31.5 GiB) is mmap'd in with
`-ngl 999`; budget at 262k ≈ 30 GiB (weights + KV + buffers), leaving ~84 `--n-gpu-layers 999`; budget at 262k ≈ 41-43 GiB (weights + KV + buffers),
GiB of aperture headroom. The 8 GiB prompt-cache budget leaving ~71 GiB of aperture headroom. The 8 GiB prompt-cache budget
(`CACHE_RAM=8192`) is the Kairic Edge qualified config. (`CACHE_RAM=8192`) is the qualified config.
Fetch: `scripts/download-frank-large-kairic.sh` (resumable, each file Fetch: `scripts/download-frank-large.sh` (per-file, each file verified
verified against its declared byte count + sha256; `--verify-only` re-checks). against its declared byte count + sha256; `--verify-only` re-checks).
Runbook: `serverAppConfigs-docs/LLM/frank-large-kairic-edge-qwen38-27b.md`. Runbook: `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md`.
Rollback: [[frank-large-unsloth-qwen38-flash-next]] (payload and runtime Rollback: [[frank-large-kairic-edge-qwen38-27b]] (payload and runtime
intact on disk). intact on disk).
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
@@ -756,7 +754,7 @@ Active:
| Service | Runbook | | Service | Runbook |
|---------|---------| |---------|---------|
| frank-large (Kairic Edge Qwen3.8-27B IU4 W4A4, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-kairic-edge-qwen38-27b.md` | | frank-large (stock llama.cpp Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md` |
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` | | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
| frank-medium (radiance TP=2 + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` | | frank-medium (radiance TP=2 + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
| frank-medium model swap (toolchain + process) | `serverAppConfigs-docs/LLM/frank-medium-model-swap.md` | | frank-medium model swap (toolchain + process) | `serverAppConfigs-docs/LLM/frank-medium-model-swap.md` |
@@ -772,6 +770,7 @@ Retired (rollback targets and history):
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` | | halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` | | CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
| unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` | | unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` |
| Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` | | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` | | vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` | | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
+3 -3
View File
@@ -267,7 +267,7 @@
"name": "frank-large", "name": "frank-large",
"category": "user", "category": "user",
"section": "AI", "section": "AI",
"description": "Qwen3.8-27B (IU4/W4A4, 26 GiB) — 262k-token context, one conversation at a time. The big brain: long, thoughtful text conversations. Text only; it cannot see images.", "description": "Qwen3.8-27B (UD-Q8_K_XL, 31.5 GiB) — 262k-token context, one conversation at a time. The big brain: long, thoughtful text conversations, DFlash2 spec decode. Text only; it cannot see images.",
"url": "", "url": "",
"docs": "/docs#models", "docs": "/docs#models",
"probe": "127.0.0.1:7770", "probe": "127.0.0.1:7770",
@@ -412,11 +412,11 @@
"category": "admin", "category": "admin",
"section": "AI", "section": "AI",
"description": "Node-based image generation.", "description": "Node-based image generation.",
"url": "", "url": "http://comfyui.franksdank.cloud:8188",
"docs": "", "docs": "",
"probe": "127.0.0.1:8188", "probe": "127.0.0.1:8188",
"method": "http", "method": "http",
"wip": false, "wip": false,
"showLink": false "showLink": true
} }
] ]