docs: update frank-large to stock llama.cpp UD-Q8_K_XL + DFlash2
- AGENTS.md: fleet topology, intro, frank-large section, text-only note, benchmarking policy, runbook index (new runbook active; Kairic Edge retired as rollback target) - homepage services.json: frank-large description -> UD-Q8_K_XL, 31.5 GiB, DFlash2 spec decode - homepage services.json: ComfyUI url + showLink (complete prior edit) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
1 parent
7689092db8
commit
600812d459
2 files changed
+38
-39
No files matched your search
@@ -12,7 +12,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
|||||||
Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service.
|
Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service.
|
||||||
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
|
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
|
||||||
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
|
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
|
||||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (Kairic Edge runtime on
|
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on
|
||||||
the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
|
the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
|
||||||
Dankclaw reverse-proxies both hosts.
|
Dankclaw reverse-proxies both hosts.
|
||||||
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
|
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
|
||||||
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
|||||||
| Target | Service | Runtime | Model | Port |
|
| Target | Service | Runtime | Model | Port |
|
||||||
|--------|---------|---------|-------|------|
|
|--------|---------|---------|-------|------|
|
||||||
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, 19.37 GiB) with the z-lab DFlash2 external drafter (depth 7), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations, UNQUANTIZED bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 from the FP8/fp8-KV shape to test whether fp8 KV had been degrading W4A8 output quality; auto-sized 486,464-token pool = 1.86x the window), prefix caching, vision | 7770 |
|
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, 19.37 GiB) with the z-lab DFlash2 external drafter (depth 7), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations, UNQUANTIZED bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 from the FP8/fp8-KV shape to test whether fp8 KV had been degrading W4A8 output quality; auto-sized 486,464-token pool = 1.86x the window), prefix caching, vision | 7770 |
|
||||||
| Dankclaw | `frank-large` | Kairic Edge (ciru-ai/ROCmFPX v1.2, gfx1151, root systemd unit, built from source) | jcbtc Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense, 26 GiB total) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), MTP depth 4 + n-gram drafting (24/64/64), compatibility mode, text only | 7770 |
|
| Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), q8_0 KV, text only | 7770 |
|
||||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
|
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
|
||||||
|
|
||||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||||
@@ -228,7 +228,7 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
|
|||||||
merely wedged kernel usually still answers ARP.
|
merely wedged kernel usually still answers ARP.
|
||||||
|
|
||||||
1. Stop and mask (self-healing would restart it):
|
1. Stop and mask (self-healing would restart it):
|
||||||
- Dankclaw `frank-large` (root unit, CIRU llama.cpp on the iGPU):
|
- Dankclaw `frank-large` (root unit, stock llama.cpp on the iGPU):
|
||||||
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
|
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
|
||||||
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
|
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
|
||||||
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
|
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
|
||||||
@@ -265,38 +265,36 @@ a role pointing at a provider the file does not define fails at call time. As
|
|||||||
of 2026-09-20 both `frank-small` and `frank-medium` accept images
|
of 2026-09-20 both `frank-small` and `frank-medium` accept images
|
||||||
(`input: [text, image]`) — frank-medium's checkpoint serves vision through the
|
(`input: [text, image]`) — frank-medium's checkpoint serves vision through the
|
||||||
Gemma4 merge path with no separate mmproj, verified on held-out synthetic
|
Gemma4 merge path with no separate mmproj, verified on held-out synthetic
|
||||||
images; `frank-large` remains text-only because its Kairic Edge deployment loads no
|
images; `frank-large` remains text-only because its stock llama.cpp deployment loads no
|
||||||
vision projector (the model has no mmproj). Note also that
|
vision projector (the mmproj is not loaded). Note also that
|
||||||
the reasoning channel differs by runtime: llama.cpp endpoints emit
|
the reasoning channel differs by runtime: llama.cpp endpoints emit
|
||||||
`reasoning_content`, vLLM emits `reasoning`.
|
`reasoning_content`, vLLM emits `reasoning`.
|
||||||
There is **no audio**
|
There is **no audio**
|
||||||
input anywhere in the fleet: that capability left with the FLM NPU endpoint on
|
input anywhere in the fleet: that capability left with the FLM NPU endpoint on
|
||||||
2026-09-11.
|
2026-09-11.
|
||||||
|
|
||||||
### frank-large — Kairic Edge Qwen3.8-27B IU4 (W4A4) (Dankclaw, ACTIVE)
|
### frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE)
|
||||||
|
|
||||||
The Kairic Edge runtime (ciru-ai/ROCmFPX, branch
|
The unsloth llama.cpp prebuilt `app-b11030-mix-5ff778e` (ROCm gfx1151,
|
||||||
`kairic-edge-qwen38-27b-v1.2`, commit `205a3e5`) as a **root systemd unit**
|
`/home/dank/GitPackages/unsloth-llama-b11030/`, bundles its own ROCm
|
||||||
(`scripts/frank-large.service`, `User=dank`), built from source
|
runtime — no host SDK gate) as a **root systemd unit**
|
||||||
(`scripts/build-kairic-edge-gfx1151.sh`, TheRock 7.15.0a20260718 toolchain,
|
(`scripts/frank-large.service`, `User=dank`), serving
|
||||||
Composable Kernel `fdf4bb7f` + the gfx1151-iu4 patch), serving
|
`unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` (27B dense, 31.5 GiB,
|
||||||
`jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` (27B dense, IU4 W4A4, 15.5 GiB main
|
sha256-verified) on port 7770 from the gfx1151 iGPU. One stream, full
|
||||||
GGUF + 7.99 GiB FFN sidecar + 1.88 GiB GDN sidecar + 0.70 GiB GDN-Output
|
262,144 ctx (the model's native window, `PARALLEL_SLOTS=1`), DFlash2 spec
|
||||||
sidecar, all sha256-verified) on port 7770 from the gfx1151 iGPU. One
|
decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, depth 4,
|
||||||
stream, full 262,144 ctx (the model's native window, `PARALLEL_SLOTS=1`),
|
`SPEC_DRAFT_N_MAX=4`), reasoning extracted to `reasoning_content`, and
|
||||||
MTP depth 4 (`SPEC_DRAFT_N_MAX=4`) + n-gram drafting (24/64/64), reasoning
|
**no vision projector**. It replaced the Kairic Edge ROCmFPX build (jcbtc
|
||||||
extracted to `reasoning_content` (compatibility mode — the harness needs
|
Qwen3.8-27B-IU4-Kairic-Edge), which had replaced the unsloth
|
||||||
sampling, tools and a reasoning budget, which fast-greedy rejects), and
|
Qwen3.8-Flash-Next UD-IQ4_XS.
|
||||||
**no vision projector**. It replaced the unsloth llama.cpp prebuilt
|
|
||||||
(b11030, Qwen3.8-Flash-Next UD-IQ4_XS, 125B MoE), which had replaced the
|
|
||||||
CIRU v4.4.1 Orca runtime.
|
|
||||||
|
|
||||||
The spec method is native MTP depth 4 + n-gram drafting. DFlash is NOT
|
The spec method is DFlash2 (block-diffusion drafter) at depth 4. The
|
||||||
available for this model (no `DRAFT_DFLASH` spec type in the Kairic Edge
|
drafter is loaded via `--model-draft`; the chain is `--spec-type
|
||||||
runtime). The three PromptForge sidecars (FFN, GDN, GDN-Output) are wired
|
draft-dflash` with `--spec-draft-device ROCm0`. KV is **q8_0** (the
|
||||||
via the `PROMPTFORGE_*` env block (the certified runner's exact env). The
|
unsloth b11030 build does not support fp8 KV; q8_0 is the
|
||||||
serve script (`scripts/serve-frank-large.sh`) validates binary → main GGUF →
|
highest-precision 8-bit option, ~8 GiB at 262k). The serve script
|
||||||
3 sidecars and exits with a pointer message if any is missing, then `exec`s
|
(`scripts/serve-frank-large.sh`) validates binary → main GGUF → drafter
|
||||||
|
and exits with a pointer message if any is missing, then `exec`s
|
||||||
`llama-server` with `--jinja --alias frank-large --chat-template-kwargs
|
`llama-server` with `--jinja --alias frank-large --chat-template-kwargs
|
||||||
'{"preserve_thinking":true}' --chat-template-file $TEMPLATE_FILE` (the
|
'{"preserve_thinking":true}' --chat-template-file $TEMPLATE_FILE` (the
|
||||||
Qwen-Sharp v22.4.0 template; without `--chat-template-file` the binary
|
Qwen-Sharp v22.4.0 template; without `--chat-template-file` the binary
|
||||||
@@ -305,16 +303,16 @@ falls back to the built-in `peg-native` template, which produces a
|
|||||||
|
|
||||||
The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
|
The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
|
||||||
`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated"
|
`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated"
|
||||||
is not the budget). The 27B dense model (26 GiB total) is mmap'd in with
|
is not the budget). The 27B dense model (31.5 GiB) is mmap'd in with
|
||||||
`-ngl 999`; budget at 262k ≈ 30 GiB (weights + KV + buffers), leaving ~84
|
`--n-gpu-layers 999`; budget at 262k ≈ 41-43 GiB (weights + KV + buffers),
|
||||||
GiB of aperture headroom. The 8 GiB prompt-cache budget
|
leaving ~71 GiB of aperture headroom. The 8 GiB prompt-cache budget
|
||||||
(`CACHE_RAM=8192`) is the Kairic Edge qualified config.
|
(`CACHE_RAM=8192`) is the qualified config.
|
||||||
|
|
||||||
Fetch: `scripts/download-frank-large-kairic.sh` (resumable, each file
|
Fetch: `scripts/download-frank-large.sh` (per-file, each file verified
|
||||||
verified against its declared byte count + sha256; `--verify-only` re-checks).
|
against its declared byte count + sha256; `--verify-only` re-checks).
|
||||||
|
|
||||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-kairic-edge-qwen38-27b.md`.
|
Runbook: `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md`.
|
||||||
Rollback: [[frank-large-unsloth-qwen38-flash-next]] (payload and runtime
|
Rollback: [[frank-large-kairic-edge-qwen38-27b]] (payload and runtime
|
||||||
intact on disk).
|
intact on disk).
|
||||||
|
|
||||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||||
@@ -756,7 +754,7 @@ Active:
|
|||||||
|
|
||||||
| Service | Runbook |
|
| Service | Runbook |
|
||||||
|---------|---------|
|
|---------|---------|
|
||||||
| frank-large (Kairic Edge Qwen3.8-27B IU4 W4A4, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-kairic-edge-qwen38-27b.md` |
|
| frank-large (stock llama.cpp Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md` |
|
||||||
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
|
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
|
||||||
| frank-medium (radiance TP=2 + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
|
| frank-medium (radiance TP=2 + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
|
||||||
| frank-medium model swap (toolchain + process) | `serverAppConfigs-docs/LLM/frank-medium-model-swap.md` |
|
| frank-medium model swap (toolchain + process) | `serverAppConfigs-docs/LLM/frank-medium-model-swap.md` |
|
||||||
@@ -772,6 +770,7 @@ Retired (rollback targets and history):
|
|||||||
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
|
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
|
||||||
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
|
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
|
||||||
| unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` |
|
| unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` |
|
||||||
|
| Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
|
||||||
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
|
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
|
||||||
| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
|
| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
|
||||||
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
||||||
|
|||||||
@@ -267,7 +267,7 @@
|
|||||||
"name": "frank-large",
|
"name": "frank-large",
|
||||||
"category": "user",
|
"category": "user",
|
||||||
"section": "AI",
|
"section": "AI",
|
||||||
"description": "Qwen3.8-27B (IU4/W4A4, 26 GiB) — 262k-token context, one conversation at a time. The big brain: long, thoughtful text conversations. Text only; it cannot see images.",
|
"description": "Qwen3.8-27B (UD-Q8_K_XL, 31.5 GiB) — 262k-token context, one conversation at a time. The big brain: long, thoughtful text conversations, DFlash2 spec decode. Text only; it cannot see images.",
|
||||||
"url": "",
|
"url": "",
|
||||||
"docs": "/docs#models",
|
"docs": "/docs#models",
|
||||||
"probe": "127.0.0.1:7770",
|
"probe": "127.0.0.1:7770",
|
||||||
@@ -412,11 +412,11 @@
|
|||||||
"category": "admin",
|
"category": "admin",
|
||||||
"section": "AI",
|
"section": "AI",
|
||||||
"description": "Node-based image generation.",
|
"description": "Node-based image generation.",
|
||||||
"url": "",
|
"url": "http://comfyui.franksdank.cloud:8188",
|
||||||
"docs": "",
|
"docs": "",
|
||||||
"probe": "127.0.0.1:8188",
|
"probe": "127.0.0.1:8188",
|
||||||
"method": "http",
|
"method": "http",
|
||||||
"wip": false,
|
"wip": false,
|
||||||
"showLink": false
|
"showLink": true
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
Reference in new issue
Block a user