diff --git a/AGENTS.md b/AGENTS.md index 69e6e8a..84544d3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,7 +12,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service. - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, - Alloy, Pi-hole) plus TWO LLM services: `frank-large` (Kairic Edge runtime on + Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on Dankclaw reverse-proxies both hosts. - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack @@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| | Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, 19.37 GiB) with the z-lab DFlash2 external drafter (depth 7), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations, UNQUANTIZED bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 from the FP8/fp8-KV shape to test whether fp8 KV had been degrading W4A8 output quality; auto-sized 486,464-token pool = 1.86x the window), prefix caching, vision | 7770 | -| Dankclaw | `frank-large` | Kairic Edge (ciru-ai/ROCmFPX v1.2, gfx1151, root systemd unit, built from source) | jcbtc Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense, 26 GiB total) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), MTP depth 4 + n-gram drafting (24/64/64), compatibility mode, text only | 7770 | +| Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), q8_0 KV, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in @@ -228,7 +228,7 @@ one: `ip neigh show ` from a peer — FAILED means off/NIC dead, while a merely wedged kernel usually still answers ARP. 1. Stop and mask (self-healing would restart it): - - Dankclaw `frank-large` (root unit, CIRU llama.cpp on the iGPU): + - Dankclaw `frank-large` (root unit, stock llama.cpp on the iGPU): `sudo systemctl stop frank-large && sudo systemctl mask frank-large` - Dankclaw `frank-small` (root unit, llama.cpp on the 3090): `sudo systemctl stop frank-small && sudo systemctl mask frank-small` @@ -265,38 +265,36 @@ a role pointing at a provider the file does not define fails at call time. As of 2026-09-20 both `frank-small` and `frank-medium` accept images (`input: [text, image]`) — frank-medium's checkpoint serves vision through the Gemma4 merge path with no separate mmproj, verified on held-out synthetic -images; `frank-large` remains text-only because its Kairic Edge deployment loads no -vision projector (the model has no mmproj). Note also that +images; `frank-large` remains text-only because its stock llama.cpp deployment loads no +vision projector (the mmproj is not loaded). Note also that the reasoning channel differs by runtime: llama.cpp endpoints emit `reasoning_content`, vLLM emits `reasoning`. There is **no audio** input anywhere in the fleet: that capability left with the FLM NPU endpoint on 2026-09-11. -### frank-large — Kairic Edge Qwen3.8-27B IU4 (W4A4) (Dankclaw, ACTIVE) +### frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE) -The Kairic Edge runtime (ciru-ai/ROCmFPX, branch -`kairic-edge-qwen38-27b-v1.2`, commit `205a3e5`) as a **root systemd unit** -(`scripts/frank-large.service`, `User=dank`), built from source -(`scripts/build-kairic-edge-gfx1151.sh`, TheRock 7.15.0a20260718 toolchain, -Composable Kernel `fdf4bb7f` + the gfx1151-iu4 patch), serving -`jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` (27B dense, IU4 W4A4, 15.5 GiB main -GGUF + 7.99 GiB FFN sidecar + 1.88 GiB GDN sidecar + 0.70 GiB GDN-Output -sidecar, all sha256-verified) on port 7770 from the gfx1151 iGPU. One -stream, full 262,144 ctx (the model's native window, `PARALLEL_SLOTS=1`), -MTP depth 4 (`SPEC_DRAFT_N_MAX=4`) + n-gram drafting (24/64/64), reasoning -extracted to `reasoning_content` (compatibility mode — the harness needs -sampling, tools and a reasoning budget, which fast-greedy rejects), and -**no vision projector**. It replaced the unsloth llama.cpp prebuilt -(b11030, Qwen3.8-Flash-Next UD-IQ4_XS, 125B MoE), which had replaced the -CIRU v4.4.1 Orca runtime. +The unsloth llama.cpp prebuilt `app-b11030-mix-5ff778e` (ROCm gfx1151, +`/home/dank/GitPackages/unsloth-llama-b11030/`, bundles its own ROCm +runtime — no host SDK gate) as a **root systemd unit** +(`scripts/frank-large.service`, `User=dank`), serving +`unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` (27B dense, 31.5 GiB, +sha256-verified) on port 7770 from the gfx1151 iGPU. One stream, full +262,144 ctx (the model's native window, `PARALLEL_SLOTS=1`), DFlash2 spec +decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, depth 4, +`SPEC_DRAFT_N_MAX=4`), reasoning extracted to `reasoning_content`, and +**no vision projector**. It replaced the Kairic Edge ROCmFPX build (jcbtc +Qwen3.8-27B-IU4-Kairic-Edge), which had replaced the unsloth +Qwen3.8-Flash-Next UD-IQ4_XS. -The spec method is native MTP depth 4 + n-gram drafting. DFlash is NOT -available for this model (no `DRAFT_DFLASH` spec type in the Kairic Edge -runtime). The three PromptForge sidecars (FFN, GDN, GDN-Output) are wired -via the `PROMPTFORGE_*` env block (the certified runner's exact env). The -serve script (`scripts/serve-frank-large.sh`) validates binary → main GGUF → -3 sidecars and exits with a pointer message if any is missing, then `exec`s +The spec method is DFlash2 (block-diffusion drafter) at depth 4. The +drafter is loaded via `--model-draft`; the chain is `--spec-type +draft-dflash` with `--spec-draft-device ROCm0`. KV is **q8_0** (the +unsloth b11030 build does not support fp8 KV; q8_0 is the +highest-precision 8-bit option, ~8 GiB at 262k). The serve script +(`scripts/serve-frank-large.sh`) validates binary → main GGUF → drafter +and exits with a pointer message if any is missing, then `exec`s `llama-server` with `--jinja --alias frank-large --chat-template-kwargs '{"preserve_thinking":true}' --chat-template-file $TEMPLATE_FILE` (the Qwen-Sharp v22.4.0 template; without `--chat-template-file` the binary @@ -305,16 +303,16 @@ falls back to the built-in `peg-native` template, which produces a The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR, `gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated" -is not the budget). The 27B dense model (26 GiB total) is mmap'd in with -`-ngl 999`; budget at 262k ≈ 30 GiB (weights + KV + buffers), leaving ~84 -GiB of aperture headroom. The 8 GiB prompt-cache budget -(`CACHE_RAM=8192`) is the Kairic Edge qualified config. +is not the budget). The 27B dense model (31.5 GiB) is mmap'd in with +`--n-gpu-layers 999`; budget at 262k ≈ 41-43 GiB (weights + KV + buffers), +leaving ~71 GiB of aperture headroom. The 8 GiB prompt-cache budget +(`CACHE_RAM=8192`) is the qualified config. -Fetch: `scripts/download-frank-large-kairic.sh` (resumable, each file -verified against its declared byte count + sha256; `--verify-only` re-checks). +Fetch: `scripts/download-frank-large.sh` (per-file, each file verified +against its declared byte count + sha256; `--verify-only` re-checks). -Runbook: `serverAppConfigs-docs/LLM/frank-large-kairic-edge-qwen38-27b.md`. -Rollback: [[frank-large-unsloth-qwen38-flash-next]] (payload and runtime +Runbook: `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md`. +Rollback: [[frank-large-kairic-edge-qwen38-27b]] (payload and runtime intact on disk). ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) @@ -756,7 +754,7 @@ Active: | Service | Runbook | |---------|---------| -| frank-large (Kairic Edge Qwen3.8-27B IU4 W4A4, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-kairic-edge-qwen38-27b.md` | +| frank-large (stock llama.cpp Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md` | | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` | | frank-medium (radiance TP=2 + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` | | frank-medium model swap (toolchain + process) | `serverAppConfigs-docs/LLM/frank-medium-model-swap.md` | @@ -772,6 +770,7 @@ Retired (rollback targets and history): | halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` | | CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` | | unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` | +| Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` | | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` | | vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` | | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` | diff --git a/shared/homepage/services.json b/shared/homepage/services.json index c0ce300..8683c39 100644 --- a/shared/homepage/services.json +++ b/shared/homepage/services.json @@ -267,7 +267,7 @@ "name": "frank-large", "category": "user", "section": "AI", - "description": "Qwen3.8-27B (IU4/W4A4, 26 GiB) — 262k-token context, one conversation at a time. The big brain: long, thoughtful text conversations. Text only; it cannot see images.", + "description": "Qwen3.8-27B (UD-Q8_K_XL, 31.5 GiB) — 262k-token context, one conversation at a time. The big brain: long, thoughtful text conversations, DFlash2 spec decode. Text only; it cannot see images.", "url": "", "docs": "/docs#models", "probe": "127.0.0.1:7770", @@ -412,11 +412,11 @@ "category": "admin", "section": "AI", "description": "Node-based image generation.", - "url": "", + "url": "http://comfyui.franksdank.cloud:8188", "docs": "", "probe": "127.0.0.1:8188", "method": "http", "wip": false, - "showLink": false + "showLink": true } ]