docs: frank-large Kairic Edge runbook + frank-medium clean-KV updates
- New runbook frank-large-kairic-edge-qwen38-27b.md (ACTIVE frank-large, Kairic Edge runtime, jcbtc Qwen3.8-27B-IU4, 262144 ctx, MTP depth 4 + n-gram drafting, compatibility mode) - AGENTS.md: frank-large section + topology row + fleet contract + runbook index (Kairic Edge ACTIVE, unsloth RETIRED same-day); frank-medium section body (MXFP4 + unquantized bf16 KV, 486,464-token pool) - LLM index + top-level index: Kairic Edge ACTIVE, unsloth RETIRED, system specs entry
This commit is contained in:
Frank committed
1 parent
b2a74004df
commit
e04f510883
4 files changed
+282
-69
No files matched your search
@@ -12,7 +12,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service.
|
||||
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
|
||||
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (unsloth llama.cpp prebuilt on
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (Kairic Edge runtime on
|
||||
the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
|
||||
Dankclaw reverse-proxies both hosts.
|
||||
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
|
||||
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, 19.37 GiB) with the z-lab DFlash2 external drafter (depth 7), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations, UNQUANTIZED bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 from the FP8/fp8-KV shape to test whether fp8 KV had been degrading W4A8 output quality; auto-sized 486,464-token pool = 1.86x the window), prefix caching, vision | 7770 |
|
||||
| Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-Flash-Next UD-IQ4_XS on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 200000 ctx single stream (1 slot), MTP depth 2 via the explicit `-md` shared-Q8_0 head, text only | 7770 |
|
||||
| Dankclaw | `frank-large` | Kairic Edge (ciru-ai/ROCmFPX v1.2, gfx1151, root systemd unit, built from source) | jcbtc Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense, 26 GiB total) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), MTP depth 4 + n-gram drafting (24/64/64), compatibility mode, text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
@@ -265,53 +265,55 @@ a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-09-20 both `frank-small` and `frank-medium` accept images
|
||||
(`input: [text, image]`) — frank-medium's checkpoint serves vision through the
|
||||
Gemma4 merge path with no separate mmproj, verified on held-out synthetic
|
||||
images; `frank-large` remains text-only because its unsloth deployment loads no
|
||||
vision projector (the F16 mmproj is on disk, one env flip away). Note also that
|
||||
images; `frank-large` remains text-only because its Kairic Edge deployment loads no
|
||||
vision projector (the model has no mmproj). Note also that
|
||||
the reasoning channel differs by runtime: llama.cpp endpoints emit
|
||||
`reasoning_content`, vLLM emits `reasoning`.
|
||||
There is **no audio**
|
||||
input anywhere in the fleet: that capability left with the FLM NPU endpoint on
|
||||
2026-09-11.
|
||||
|
||||
### frank-large — unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS (Dankclaw, ACTIVE)
|
||||
### frank-large — Kairic Edge Qwen3.8-27B IU4 (W4A4) (Dankclaw, ACTIVE)
|
||||
|
||||
The unsloth llama.cpp prebuilt `app-b11030-mix-5ff778e` (ROCm gfx1151,
|
||||
`/home/dank/GitPackages/unsloth-llama-b11030/`, bundles its own ROCm runtime —
|
||||
no host SDK gate, no venv) as a **root systemd unit** (`scripts/
|
||||
frank-large.service`, `User=dank`), serving
|
||||
`unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (87.3 GiB main GGUF as three HF
|
||||
split parts, loaded natively as a split via `-m` on part 1, + a 2.6 GiB MTP head, all sha256-verified) on port
|
||||
7770 from the gfx1151 iGPU. One stream, 200,000 ctx (`PARALLEL_SLOTS=1`,
|
||||
operator's requested shape; the trained window is 262,144), MTP depth 2
|
||||
(`SPEC_DRAFT_N_MAX=2`, the model card's default), reasoning extracted to
|
||||
`reasoning_content`, and **no vision projector** (the F16 mmproj is on disk,
|
||||
`ENABLE_VISION=0` — one env flip to enable). It replaced the CIRU v4.4.1
|
||||
Orca runtime, which had replaced halogen-flash-server.
|
||||
The Kairic Edge runtime (ciru-ai/ROCmFPX, branch
|
||||
`kairic-edge-qwen38-27b-v1.2`, commit `205a3e5`) as a **root systemd unit**
|
||||
(`scripts/frank-large.service`, `User=dank`), built from source
|
||||
(`scripts/build-kairic-edge-gfx1151.sh`, TheRock 7.15.0a20260718 toolchain,
|
||||
Composable Kernel `fdf4bb7f` + the gfx1151-iu4 patch), serving
|
||||
`jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` (27B dense, IU4 W4A4, 15.5 GiB main
|
||||
GGUF + 7.99 GiB FFN sidecar + 1.88 GiB GDN sidecar + 0.70 GiB GDN-Output
|
||||
sidecar, all sha256-verified) on port 7770 from the gfx1151 iGPU. One
|
||||
stream, full 262,144 ctx (the model's native window, `PARALLEL_SLOTS=1`),
|
||||
MTP depth 4 (`SPEC_DRAFT_N_MAX=4`) + n-gram drafting (24/64/64), reasoning
|
||||
extracted to `reasoning_content` (compatibility mode — the harness needs
|
||||
sampling, tools and a reasoning budget, which fast-greedy rejects), and
|
||||
**no vision projector**. It replaced the unsloth llama.cpp prebuilt
|
||||
(b11030, Qwen3.8-Flash-Next UD-IQ4_XS, 125B MoE), which had replaced the
|
||||
CIRU v4.4.1 Orca runtime.
|
||||
|
||||
The runtime is the first with an MTP graph for the qwen4exp architecture
|
||||
(stock ggml-org mainline has none), so the unsloth prebuilt is the
|
||||
llama.cpp-compatible choice. The MTP head is loaded via an **explicit
|
||||
`-md`**: sidecar auto-discovery does not search the `mtp/` subfolder, and
|
||||
without it `--spec-type draft-mtp` silently runs at the main model's speed.
|
||||
The serve script (`scripts/serve-frank-large.sh`) validates binary → main
|
||||
GGUF → MTP head and exits with a pointer message if any is missing, then
|
||||
`exec`s `llama-server` with `--jinja --alias frank-large --chat-template-
|
||||
kwargs '{"preserve_thinking":true}'` (Qwen3.8's template needs this to
|
||||
carry thinking blocks across turns; a request-level value still wins).
|
||||
The spec method is native MTP depth 4 + n-gram drafting. DFlash is NOT
|
||||
available for this model (no `DRAFT_DFLASH` spec type in the Kairic Edge
|
||||
runtime). The three PromptForge sidecars (FFN, GDN, GDN-Output) are wired
|
||||
via the `PROMPTFORGE_*` env block (the certified runner's exact env). The
|
||||
serve script (`scripts/serve-frank-large.sh`) validates binary → main GGUF →
|
||||
3 sidecars and exits with a pointer message if any is missing, then `exec`s
|
||||
`llama-server` with `--jinja --alias frank-large --chat-template-kwargs
|
||||
'{"preserve_thinking":true}'` (Qwen3.8's template needs this to carry
|
||||
thinking blocks across turns; a request-level value still wins).
|
||||
|
||||
The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
|
||||
`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated"
|
||||
is not the budget). The 87.3 GiB model is mmap'd in with `-ngl 999`; budget
|
||||
at 200k ≈ 100 GiB (weights + MTP head + KV + buffers), leaving ~14 GiB of
|
||||
aperture headroom. The n-gram embedding (PLE) is inside the GGUF — no
|
||||
external PLE payload or pager, unlike the CIRU Orca packaging.
|
||||
is not the budget). The 27B dense model (26 GiB total) is mmap'd in with
|
||||
`-ngl 999`; budget at 262k ≈ 30 GiB (weights + KV + buffers), leaving ~84
|
||||
GiB of aperture headroom. The 8 GiB prompt-cache budget
|
||||
(`CACHE_RAM=8192`) is the Kairic Edge qualified config.
|
||||
|
||||
Fetch: `scripts/download-frank-large-unsloth.sh` (resumable, each part
|
||||
verified against its LFS sha256 fetched live from the Hub tree API, then the
|
||||
assembled GGUF against `SHA256SUMS`; `--verify-only` re-checks).
|
||||
Fetch: `scripts/download-frank-large-kairic.sh` (resumable, each file
|
||||
verified against its declared byte count + sha256; `--verify-only` re-checks).
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md`.
|
||||
Rollback: [[frank-large-ciru-orca]] (payload and runtime intact on disk).
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-kairic-edge-qwen38-27b.md`.
|
||||
Rollback: [[frank-large-unsloth-qwen38-flash-next]] (payload and runtime
|
||||
intact on disk).
|
||||
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
@@ -343,19 +345,19 @@ Rollback: [[frank-small-qwen38-gsq-rco]].
|
||||
|
||||
vllm-radiance **0.9.3** (`localhost/radiance-vllm:0.9.3`, rootless **Quadlet**
|
||||
`dankserver/frank-medium/quadlet/frank-medium.container`) serving
|
||||
`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, dynamic
|
||||
activation quant, 30.9 GB / 7 shards) with the **z-lab DFlash2 external
|
||||
drafter** (`num_speculative_tokens: 7`, live mean accept 4.24/7) in **one
|
||||
tensor-parallel process across both R9700s** (`--tensor-parallel-size 2`).
|
||||
262,144 ctx, `--max-num-seqs 8`, chunk 8192, fp8 KV, prefix caching, GPU fill
|
||||
0.98, **no KV-memory pin** — the pool auto-sized to 384,238 tokens (1.47× the
|
||||
full window; the DFlash2 drafter's 2.1 GiB costs the pool vs MTP's 461,540).
|
||||
DFlash2 was A/B tested against MTP self-speculation on this checkpoint and
|
||||
wins on decode (100.0 vs 85.9 t/s) — see the runbook. Tools via `qwen3_coder`,
|
||||
reasoning via `qwen3` (the field is `reasoning`, not llama.cpp's
|
||||
`reasoning_content`), vision through the Gemma4 merge path, chat template
|
||||
Qwen-Sharp v22.4.0 (froggeric, repo-tracked at `dankserver/frank-medium/`,
|
||||
terse system prompt by default).
|
||||
`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4/quark W4A8, 19.37
|
||||
GiB) with the **z-lab DFlash2 external drafter** (`num_speculative_tokens: 7`,
|
||||
live mean accept 4.24/7) in **one tensor-parallel process across both
|
||||
R9700s** (`--tensor-parallel-size 2`). 262,144 ctx, `--max-num-seqs 8`, chunk
|
||||
8192, **unquantized bf16 KV** (`KV_DTYPE=auto`, cut 2026-09-21 from the
|
||||
FP8/fp8-KV shape to test whether fp8 KV had been degrading W4A8 output
|
||||
quality), prefix caching, GPU fill 0.98, **no KV-memory pin** — the pool
|
||||
auto-sized to 486,464 tokens (1.86× the full window). DFlash2 was A/B tested
|
||||
against MTP self-speculation on this checkpoint and wins on decode (100.0 vs
|
||||
85.9 t/s) — see the runbook. Tools via `qwen3_coder`, reasoning via `qwen3`
|
||||
(the field is `reasoning`, not llama.cpp's `reasoning_content`), vision through
|
||||
the Gemma4 merge path, chat template Qwen-Sharp v22.4.0 (froggeric,
|
||||
repo-tracked at `dankserver/frank-medium/`, terse system prompt by default).
|
||||
|
||||
**The entrypoint is quantization-aware.** `radiance-entrypoint.sh` reads
|
||||
`quantization_config.quant_method` from the checkpoint's `config.json` and
|
||||
@@ -752,7 +754,7 @@ Active:
|
||||
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md` |
|
||||
| frank-large (Kairic Edge Qwen3.8-27B IU4 W4A4, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-kairic-edge-qwen38-27b.md` |
|
||||
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
|
||||
| frank-medium (radiance TP=2 + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
|
||||
| frank-medium model swap (toolchain + process) | `serverAppConfigs-docs/LLM/frank-medium-model-swap.md` |
|
||||
@@ -767,6 +769,7 @@ Retired (rollback targets and history):
|
||||
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
|
||||
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
|
||||
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
|
||||
| unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` |
|
||||
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
|
||||
| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
|
||||
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
||||
|
||||
Reference in new issue
Block a user