docs: update frank-large to stock llama.cpp UD-Q8_K_XL + DFlash2
- AGENTS.md: fleet topology, intro, frank-large section, text-only note, benchmarking policy, runbook index (new runbook active; Kairic Edge retired as rollback target) - homepage services.json: frank-large description -> UD-Q8_K_XL, 31.5 GiB, DFlash2 spec decode - homepage services.json: ComfyUI url + showLink (complete prior edit) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Frank
Claude
@@ -12,7 +12,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service.
|
||||
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
|
||||
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (Kairic Edge runtime on
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on
|
||||
the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
|
||||
Dankclaw reverse-proxies both hosts.
|
||||
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
|
||||
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, 19.37 GiB) with the z-lab DFlash2 external drafter (depth 7), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations, UNQUANTIZED bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 from the FP8/fp8-KV shape to test whether fp8 KV had been degrading W4A8 output quality; auto-sized 486,464-token pool = 1.86x the window), prefix caching, vision | 7770 |
|
||||
| Dankclaw | `frank-large` | Kairic Edge (ciru-ai/ROCmFPX v1.2, gfx1151, root systemd unit, built from source) | jcbtc Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense, 26 GiB total) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), MTP depth 4 + n-gram drafting (24/64/64), compatibility mode, text only | 7770 |
|
||||
| Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), q8_0 KV, text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
@@ -228,7 +228,7 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
|
||||
merely wedged kernel usually still answers ARP.
|
||||
|
||||
1. Stop and mask (self-healing would restart it):
|
||||
- Dankclaw `frank-large` (root unit, CIRU llama.cpp on the iGPU):
|
||||
- Dankclaw `frank-large` (root unit, stock llama.cpp on the iGPU):
|
||||
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
|
||||
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
|
||||
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
|
||||
@@ -265,38 +265,36 @@ a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-09-20 both `frank-small` and `frank-medium` accept images
|
||||
(`input: [text, image]`) — frank-medium's checkpoint serves vision through the
|
||||
Gemma4 merge path with no separate mmproj, verified on held-out synthetic
|
||||
images; `frank-large` remains text-only because its Kairic Edge deployment loads no
|
||||
vision projector (the model has no mmproj). Note also that
|
||||
images; `frank-large` remains text-only because its stock llama.cpp deployment loads no
|
||||
vision projector (the mmproj is not loaded). Note also that
|
||||
the reasoning channel differs by runtime: llama.cpp endpoints emit
|
||||
`reasoning_content`, vLLM emits `reasoning`.
|
||||
There is **no audio**
|
||||
input anywhere in the fleet: that capability left with the FLM NPU endpoint on
|
||||
2026-09-11.
|
||||
|
||||
### frank-large — Kairic Edge Qwen3.8-27B IU4 (W4A4) (Dankclaw, ACTIVE)
|
||||
### frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE)
|
||||
|
||||
The Kairic Edge runtime (ciru-ai/ROCmFPX, branch
|
||||
`kairic-edge-qwen38-27b-v1.2`, commit `205a3e5`) as a **root systemd unit**
|
||||
(`scripts/frank-large.service`, `User=dank`), built from source
|
||||
(`scripts/build-kairic-edge-gfx1151.sh`, TheRock 7.15.0a20260718 toolchain,
|
||||
Composable Kernel `fdf4bb7f` + the gfx1151-iu4 patch), serving
|
||||
`jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` (27B dense, IU4 W4A4, 15.5 GiB main
|
||||
GGUF + 7.99 GiB FFN sidecar + 1.88 GiB GDN sidecar + 0.70 GiB GDN-Output
|
||||
sidecar, all sha256-verified) on port 7770 from the gfx1151 iGPU. One
|
||||
stream, full 262,144 ctx (the model's native window, `PARALLEL_SLOTS=1`),
|
||||
MTP depth 4 (`SPEC_DRAFT_N_MAX=4`) + n-gram drafting (24/64/64), reasoning
|
||||
extracted to `reasoning_content` (compatibility mode — the harness needs
|
||||
sampling, tools and a reasoning budget, which fast-greedy rejects), and
|
||||
**no vision projector**. It replaced the unsloth llama.cpp prebuilt
|
||||
(b11030, Qwen3.8-Flash-Next UD-IQ4_XS, 125B MoE), which had replaced the
|
||||
CIRU v4.4.1 Orca runtime.
|
||||
The unsloth llama.cpp prebuilt `app-b11030-mix-5ff778e` (ROCm gfx1151,
|
||||
`/home/dank/GitPackages/unsloth-llama-b11030/`, bundles its own ROCm
|
||||
runtime — no host SDK gate) as a **root systemd unit**
|
||||
(`scripts/frank-large.service`, `User=dank`), serving
|
||||
`unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` (27B dense, 31.5 GiB,
|
||||
sha256-verified) on port 7770 from the gfx1151 iGPU. One stream, full
|
||||
262,144 ctx (the model's native window, `PARALLEL_SLOTS=1`), DFlash2 spec
|
||||
decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, depth 4,
|
||||
`SPEC_DRAFT_N_MAX=4`), reasoning extracted to `reasoning_content`, and
|
||||
**no vision projector**. It replaced the Kairic Edge ROCmFPX build (jcbtc
|
||||
Qwen3.8-27B-IU4-Kairic-Edge), which had replaced the unsloth
|
||||
Qwen3.8-Flash-Next UD-IQ4_XS.
|
||||
|
||||
The spec method is native MTP depth 4 + n-gram drafting. DFlash is NOT
|
||||
available for this model (no `DRAFT_DFLASH` spec type in the Kairic Edge
|
||||
runtime). The three PromptForge sidecars (FFN, GDN, GDN-Output) are wired
|
||||
via the `PROMPTFORGE_*` env block (the certified runner's exact env). The
|
||||
serve script (`scripts/serve-frank-large.sh`) validates binary → main GGUF →
|
||||
3 sidecars and exits with a pointer message if any is missing, then `exec`s
|
||||
The spec method is DFlash2 (block-diffusion drafter) at depth 4. The
|
||||
drafter is loaded via `--model-draft`; the chain is `--spec-type
|
||||
draft-dflash` with `--spec-draft-device ROCm0`. KV is **q8_0** (the
|
||||
unsloth b11030 build does not support fp8 KV; q8_0 is the
|
||||
highest-precision 8-bit option, ~8 GiB at 262k). The serve script
|
||||
(`scripts/serve-frank-large.sh`) validates binary → main GGUF → drafter
|
||||
and exits with a pointer message if any is missing, then `exec`s
|
||||
`llama-server` with `--jinja --alias frank-large --chat-template-kwargs
|
||||
'{"preserve_thinking":true}' --chat-template-file $TEMPLATE_FILE` (the
|
||||
Qwen-Sharp v22.4.0 template; without `--chat-template-file` the binary
|
||||
@@ -305,16 +303,16 @@ falls back to the built-in `peg-native` template, which produces a
|
||||
|
||||
The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
|
||||
`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated"
|
||||
is not the budget). The 27B dense model (26 GiB total) is mmap'd in with
|
||||
`-ngl 999`; budget at 262k ≈ 30 GiB (weights + KV + buffers), leaving ~84
|
||||
GiB of aperture headroom. The 8 GiB prompt-cache budget
|
||||
(`CACHE_RAM=8192`) is the Kairic Edge qualified config.
|
||||
is not the budget). The 27B dense model (31.5 GiB) is mmap'd in with
|
||||
`--n-gpu-layers 999`; budget at 262k ≈ 41-43 GiB (weights + KV + buffers),
|
||||
leaving ~71 GiB of aperture headroom. The 8 GiB prompt-cache budget
|
||||
(`CACHE_RAM=8192`) is the qualified config.
|
||||
|
||||
Fetch: `scripts/download-frank-large-kairic.sh` (resumable, each file
|
||||
verified against its declared byte count + sha256; `--verify-only` re-checks).
|
||||
Fetch: `scripts/download-frank-large.sh` (per-file, each file verified
|
||||
against its declared byte count + sha256; `--verify-only` re-checks).
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-kairic-edge-qwen38-27b.md`.
|
||||
Rollback: [[frank-large-unsloth-qwen38-flash-next]] (payload and runtime
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md`.
|
||||
Rollback: [[frank-large-kairic-edge-qwen38-27b]] (payload and runtime
|
||||
intact on disk).
|
||||
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
@@ -756,7 +754,7 @@ Active:
|
||||
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (Kairic Edge Qwen3.8-27B IU4 W4A4, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-kairic-edge-qwen38-27b.md` |
|
||||
| frank-large (stock llama.cpp Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md` |
|
||||
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
|
||||
| frank-medium (radiance TP=2 + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
|
||||
| frank-medium model swap (toolchain + process) | `serverAppConfigs-docs/LLM/frank-medium-model-swap.md` |
|
||||
@@ -772,6 +770,7 @@ Retired (rollback targets and history):
|
||||
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
|
||||
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
|
||||
| unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` |
|
||||
| Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
|
||||
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
|
||||
| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
|
||||
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
||||
|
||||
@@ -267,7 +267,7 @@
|
||||
"name": "frank-large",
|
||||
"category": "user",
|
||||
"section": "AI",
|
||||
"description": "Qwen3.8-27B (IU4/W4A4, 26 GiB) — 262k-token context, one conversation at a time. The big brain: long, thoughtful text conversations. Text only; it cannot see images.",
|
||||
"description": "Qwen3.8-27B (UD-Q8_K_XL, 31.5 GiB) — 262k-token context, one conversation at a time. The big brain: long, thoughtful text conversations, DFlash2 spec decode. Text only; it cannot see images.",
|
||||
"url": "",
|
||||
"docs": "/docs#models",
|
||||
"probe": "127.0.0.1:7770",
|
||||
@@ -412,11 +412,11 @@
|
||||
"category": "admin",
|
||||
"section": "AI",
|
||||
"description": "Node-based image generation.",
|
||||
"url": "",
|
||||
"url": "http://comfyui.franksdank.cloud:8188",
|
||||
"docs": "",
|
||||
"probe": "127.0.0.1:8188",
|
||||
"method": "http",
|
||||
"wip": false,
|
||||
"showLink": false
|
||||
"showLink": true
|
||||
}
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user