From 764cf41222e51a05cd3c921d0f0c4d42cc2c4005 Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 19 Sep 2026 06:12:33 -0700 Subject: [PATCH] docs: wrap up the 2026-09-19 fleet cutover frank-large -> CIRU v4.4.1 Orca (new runbook), frank-medium -> the deployed orcarouter Q8_0 / DFlash2 Q4_K_M / balanced 32:33 / 327680 state. The d374ab9 frank-medium deploy updated only scripts/tests and left AGENTS.md, the runbook and both indexes on the predecessor (ajgazin Swift UD-Q8_K_XL, 35:30, 393216). - AGENTS.md: frank-large section (CIRU Orca, root unit, runtime gate, PLE, MTP) + frank-medium section and topology row to the deployed shape - new runbook frank-large-ciru-orca.md - frank-medium runbook header/intro: orcarouter Q8_0 swap, 32:33, 327680; predecessor kept as calibration history - LLM index.md: intro + mermaid (halogen->CIRU, GSQ-RCO->Gemma4, Swift->orcarouter), add ciru-orca, retire halogen - top-level index.md: add ciru-orca link (docs-index seam), retire halogen, fix frank-medium entry --- AGENTS.md | 122 +++++----- .../LLM/frank-large-ciru-orca.md | 219 ++++++++++++++++++ .../LLM/frank-medium-llamacpp-dflash2.md | 19 +- serverAppConfigs-docs/LLM/index.md | 58 ++--- serverAppConfigs-docs/index.md | 28 ++- 5 files changed, 340 insertions(+), 106 deletions(-) create mode 100644 serverAppConfigs-docs/LLM/frank-large-ciru-orca.md diff --git a/AGENTS.md b/AGENTS.md index 227ef1f..d623343 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,7 +12,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service. - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, - Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen-flash-server on + Alloy, Pi-hole) plus TWO LLM services: `frank-large` (CIRU v4.4.1 llama.cpp on the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on Dankclaw reverse-proxies both hosts. - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack @@ -24,8 +24,8 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (35:30 — skewed toward the x16-slot card; the other sits behind a Gen4 x1 chipset path), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj)|7770| -| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 | +|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|orcarouter Qwen3.8-27B-Uncensored Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), 2x Radeon AI PRO R9700 in ONE layer-split process (32:33 balanced; ROCm0 Gen4 x16, ROCm1 Gen4 x1 chipset path), 327680 ctx over 2 slots of 163840, batch 6144x512, f16 KV + 16 GiB host-RAM prompt cache, vision (BF16 mmproj)|7770| +| Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in @@ -68,8 +68,8 @@ All edits flow through git. Never edit files on a remote machine directly. The planner maps changed paths to services; the remote runner restarts only the selected service and records its revision outside the checkout. Documentation-only changes are a no-op. LLM services are selected only by -their own files — frank-large by `dankclaw/frank-large/quadlet/`, frank-small -and frank-medium by their unit/env/serve trio in `scripts/` — so unrelated +their own files — all three by their unit/env/serve trio in `scripts/` — so +unrelated container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless runs on a host without passwordless sudo; the value never enters the repo. @@ -207,9 +207,8 @@ archive. **STOP the production service BEFORE benchmarking** — these servers serve real users, and a bench beside a live service starves the GPU and corrupts -results. Use the runtime's own harness: `llama-bench` (llama.cpp), `vllm -bench` (vLLM), and halogen-flash-server's own `bench`/`sweep` modes for -frank-large, which is the active runtime there again. Rolling your own bench +results. Use the runtime's own harness: `llama-bench` for every llama.cpp +endpoint (all three are llama.cpp now). Rolling your own bench requires asking first. Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service @@ -227,11 +226,8 @@ one: `ip neigh show ` from a peer — FAILED means off/NIC dead, while a merely wedged kernel usually still answers ARP. 1. Stop and mask (self-healing would restart it): - - Dankclaw `frank-large` (rootless Quadlet again since 2026-09-16): - `systemctl --user stop frank-large`, then move its - `~/.config/containers/systemd/frank-large.container` aside and - `systemctl --user daemon-reload` — the quadlet generator re-creates that - unit from the file on every reload, so the source is the lever, not `mask`. + - Dankclaw `frank-large` (root unit, CIRU llama.cpp on the iGPU): + `sudo systemctl stop frank-large && sudo systemctl mask frank-large` - Dankclaw `frank-small` (root unit, llama.cpp on the 3090): `sudo systemctl stop frank-small && sudo systemctl mask frank-small` - Dankserver `frank-medium` (system unit): @@ -264,42 +260,45 @@ provider block, model `id`, `contextWindow`, `maxTokens`, `input` in `models.yml`, then fix every `modelRoles` entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As of 2026-09-16 the image path is `frank-small` (`input: [text, image]`), and -`frank-large` is text-only because its halogen deployment loads no vision -tower. There is **no audio** input anywhere in the fleet: that capability left -with the FLM NPU endpoint on 2026-09-11. +`frank-large` is text-only because its CIRU deployment loads no vision +projector (the F16 mmproj is on disk, one env flip away). There is **no audio** +input anywhere in the fleet: that capability left with the FLM NPU endpoint on +2026-09-11. -### frank-large — halogen-flash-server (Dankclaw, ACTIVE) +### frank-large — CIRU v4.4.1 Orca (Dankclaw, ACTIVE) -halogen-flash-server 0.5.8 as a **rootless Quadlet again since 2026-09-16** -(`dankclaw/frank-large/quadlet/frank-large.container`), serving -`peonist-ai/halogen-qwen3.8-flash-next` (115.55 GiB w4b `.hgn` checkpoint + a -quality overlay that re-fetched to 2.40 GiB on 2026-09-16) on port 7770 from -the gfx1151 iGPU. Two conversations of 200,192 tokens resident (pool 400,384, -2 slots), greedy default, reasoning extracted to `reasoning_content`, and **no -vision tower** — that moved to frank-small. It replaced ik_llama.cpp, which had -replaced it for two days. +CIRU v4.4.1 (the QSA window hotfix) as a **root systemd unit** (`scripts/ +frank-large.service`, `User=dank`), serving `jcbtc/Qwen3.8-Flash-CIRU-STRIX- +Orca` (79.4 GiB main GGUF + a 4.1 GiB MTP head + a 52.4 GiB PLE payload, all +sha256-verified) on port 7770 from the gfx1151 iGPU. One stream, the model's +FULL 262,144 trained window (`PARALLEL_SLOTS=1`; the MTP drafter needs exactly +one slot in this runtime), MTP depth 4, PLE cache 4096 MiB, reasoning extracted +to `reasoning_content`, and **no vision projector** (the F16 mmproj is on disk, +`ENABLE_VISION=0` — one env flip to enable). It replaced halogen-flash-server, +which had replaced ik_llama.cpp. -The engine **fetches its own weights on first start** (`HALOGEN_DOWNLOAD`, into -the `/home/dank/models/halogen` bind mount): clearing that directory and -restarting the unit is the supported way to pick up upstream updates, and it -needs the directory to exist **empty** — a missing directory fails the -container with `Error: statfs /home/dank/models/halogen: no such file or -directory` and the unit loops on `Restart=on-failure` until `mkdir -p` brings -it back. The sidecars are versioned in place, so a stale copy is invisible from -the file name (both overlays were ~95 MB larger upstream than the copies this -host had been serving). +The runtime is a **private host build** (`/home/dank/GitPackages/ciru-runtime- +v4.4.1-qsa-hotfix/`), not a container image: a gfx1151 HIP build of the CIRU +llama.cpp fork with its own bundled ROCm SDK in a venv (`.venv-rocm/`), and the +binary resolves the SDK libs via RUNPATH. The launcher (`scripts/ciru/ +run-orca-server.sh` → `run-server.sh`) has a **runtime gate**: it exits 2 unless +`${repo_root}/runtime/hip/lib/libamdhip64.so` and `runtime/rocr/lib/ +libhsa-runtime64.so` exist. The serve script (`scripts/serve-frank-large.sh`) +satisfies that gate idempotently by `ln -sfn`-ing both to the venv's unversioned +devel symlinks, then `exec`s the launcher with `--chat-template-kwargs +'{"preserve_thinking":true}'` (Qwen3.8's template needs this to carry thinking +blocks across turns; a request-level value still wins). -Measured boot ledger (2026-09-16, `HALOGEN_VERBOSE=1` ships in the unit): -device budget `121 − 67.7 − 20 = 33.5 GiB` against a need of `~27.2 GiB`, so -the configured pool was **not clamped**; 68.0 GiB of weights locked + 11.0 GiB -of KV pool + 16.5 GiB of working memory = 95.4 GiB in all, leaving 14.7 GiB -(475 contiguous 2 MiB blocks) for everything else: frank-small on the 3090 and -~28 co-tenant containers. The engine printed its co-tenant stall NOTE and -logged 4,350 compaction stalls (285 failed) while reserving the pool, so read -that as "tight and fragmented", not "out of memory". +The serve script validates launcher → built binary → venv runtime → main GGUF → +MTP head → PLE payload + manifest + scale, and exits with a pointer message if +any is missing. `PLE_CACHE_MIB=4096` is the vendor production-profile default; +the 52.4 GiB PLE is mmap'd and paged, not resident. -Runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`. -Rollback: [[frank-large-ik-uncensored]]. +The halogen Quadlet (`dankclaw/frank-large/quadlet/`) was retired to +`.AGENTTRASH/` — one owner per endpoint, now the scripts/ trio. + +Runbook: `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md`. +Rollback: [[halogen-flash-server]]. ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) @@ -327,25 +326,25 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`. Rollback: [[frank-small-qwen38-gsq-rco]]. -### frank-medium — llama.cpp layer split, MTP drafter (Dankserver, ACTIVE) +### frank-medium — llama.cpp layer split, DFlash2 drafter (Dankserver, ACTIVE) Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root systemd unit `scripts/frank-medium.service`) serving -`ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL** -(29.29 GiB) with the model's OWN **MTP head** (`blk..nextn.*` in the file; -the z-lab DFlash2 `Q8_0` sidecar stays on disk and in the fetch script — one -`SPEC_TYPE` change brings the faster chain back), in **one process across both -R9700s**, `--split-mode layer` at a **35:30 tensor split** — one layer moved -to ROCm0 on 2026-09-17 because the cards' PCIe paths are not symmetric: -ROCm0 (03:00.0) is CPU-root-port Gen4 x16 (~31.5 GB/s) while ROCm1 -(0b:00.0) rides the B650 chipset at Gen4 x1 (~1.97 GB/s), and with the -drafter, output layers and heavier compute buffers pinned to the -last-layers card, the bottlenecked card was running 98% while the x16 card -sat at 40%. 36:29 would cross a full-attention layer and fails both the -1 GiB floor and the fit slack, so 35:30 is the farthest safe shift at this -pool. 393216 ctx over **2 slots** of 196608, batch 6144x512, vision -via the BF16 mmproj. DSpark was -put up in the drafter's place the same day and reverted: it declares no sliding +`orcarouter/Qwen3.8-27B-Uncensored-GGUF` at **Q8_0** (27.0 GiB) with the +**z-lab DFlash2 `Q4_K_M` drafter** (1.06 GiB). The target file also carries the +model's own MTP head (`blk..nextn.*`, `nextn_predict_layers=1`), so +`draft-mtp` is one `SPEC_TYPE` change away — the drafter file stays on disk. In +**one process across both R9700s**, `--split-mode layer` at a **32:33 tensor +split** — balanced (operator call 2026-09-19, the most even split 65 layers +allow; ROCm1, the last-layers card that also carries the drafter and output +layers, holds the extra one). The predecessor's 35:30 skew (2026-09-17) leaned +work onto the x16 card because ROCm1 (0b:00.0) rides the B650 chipset at Gen4 +x1 (~1.97 GB/s) while ROCm0 (03:00.0) is CPU-root-port Gen4 x16 (~31.5 GB/s); +the balanced point trades that lean for symmetry and was accepted. 327680 ctx +over **2 slots** of 163840 — inside the checkpoint's trained 262144 window, so +no rope extrapolation (unlike the predecessor's 393216) — batch 6144x512, f16 +KV, a 16384 MiB host-RAM prompt cache, vision via the BF16 mmproj. DSpark was +tried in the drafter's place on 2026-09-16 and reverted: it declares no sliding window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four layers of the split. @@ -706,7 +705,7 @@ Active: | Service | Runbook | |---------|---------| -| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` | +| frank-large (CIRU v4.4.1 Orca, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` | | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` | | frank-medium (llama.cpp + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | @@ -718,6 +717,7 @@ Retired (rollback targets and history): |---------|------|--------|---------| | kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` | | ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` | +| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` | | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` | | vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` | | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` | diff --git a/serverAppConfigs-docs/LLM/frank-large-ciru-orca.md b/serverAppConfigs-docs/LLM/frank-large-ciru-orca.md new file mode 100644 index 0000000..623d85d --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-large-ciru-orca.md @@ -0,0 +1,219 @@ +--- +type: reference +title: CIRU v4.4.1 Orca on Dankclaw (frank-large) +description: The active frank-large runbook — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing. +tags: + - llm + - ciru + - dankclaw + - frank-large + - strix-halo + - gfx1151 + - qwen4exp + - rocm + - mtp +timestamp: 2026-09-19 +--- + +# CIRU v4.4.1 Orca on Dankclaw (frank-large) + +> **ACTIVE since 2026-09-19.** It replaced halogen-flash-server (see +> [[halogen-flash-server]], the current rollback), which had replaced +> ik_llama.cpp (see [[frank-large-ik-uncensored]]). Deployed as a **root +> systemd unit** (`scripts/frank-large.service`, `User=dank`), port 7770, on the +> same gfx1151 iGPU the halogen quadlet used. The RTX 3090 it does not touch +> belongs to [[frank-small-gemma4-12b]]. + +CIRU is a llama.cpp fork tuned for the Qwen3.8-Flash-Next ("qwen4exp") +architecture on AMD gfx1151 (Strix Halo). v4.4.1 carries a **QSA window +hotfix** that is what makes vision and the MTP drafter work on this build; the +pre-applied hotfix source ships in the model repo under +`runtime/v4.4.1/qsa-window-hotfix/`. Like every other fleet endpoint it is a +llama.cpp runtime — `llama-bench` is its own harness. + +## Shape + +One root unit, one process, one published port: + +| half | port | reachable | +|---|---|---| +| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** | + +Unlike the halogen predecessor (two ports: an unauthenticated engine on 8730 +and an OpenAI front-end on 8731), CIRU is a single `llama-server` process — +there is no separate engine port to keep private. + +`/v1/models` reports the loaded model name, **`Qwen3.8-Flash-CIRU-STRIX-Orca`** +(the launcher sets no `--alias`). The endpoint serves **whatever model name a +request names** — llama.cpp does not reject an unknown id — so the harness's +`frank-large` label in `models.yml` works without a rename. + +## Runtime + +A **private host build**, not a container image. The v4.4.1 source with the QSA +window hotfix, built on the host against the private ROCm SDK that +`scripts/ciru/setup-linux-amd.sh` installs into the checkout's own venv: + +| path | what | +|---|---| +| `/home/dank/GitPackages/ciru-runtime-v4.4.1-qsa-hotfix/` | the checkout (source + build + venv) | +| `build-gfx1151-sdk/bin/llama-server` | the built binary | +| `.venv-rocm/` | the private ROCm SDK venv (~5.7 GB); the binary resolves its libs through RUNPATH | +| `scripts/ciru/run-orca-server.sh` | the launcher (`exec`'d by the serve script) | +| `profiles/strix-halo-production.env` | the 36-variable kernel bundle + PLE pager defaults, sourced by the launcher AFTER `frank-large.env` (every line is `${VAR:-default}`, so a value set in `frank-large.env` wins) | + +### The runtime gate + +`run-orca-server.sh` sources `run-server.sh`, which **exits 2** unless +`${repo_root}/runtime/hip/lib/libamdhip64.so` and `runtime/rocr/lib/ +libhsa-runtime64.so` exist (unversioned, no suffix). It prepends both to +`LD_LIBRARY_PATH` and `ldd`-checks that the built `libllama-common.so` carries +the "Qwen4Exp MTP image position" marker and the HIP lib carries the +`flash_attn_index_mask_clear/set/empty` symbols before exec. + +The build links the venv through RUNPATH, so those two files are an existence +gate, not a real dependency. `serve-frank-large.sh` satisfies the gate +idempotently: it finds the venv's unversioned devel symlinks +(`.venv-rocm/lib/python*/site-packages/_rocm_sdk_devel/lib/`) and `ln -sfn`-s +both into `runtime/{hip,rocr}/lib/`. **The first deploy of this endpoint failed +exactly here** — the build linked the venv but `runtime/` did not exist, so the +launcher exited 2 and the unit crash-looped until the serve script created the +symlinks. + +## Payload + +`scripts/download-frank-large-orca.sh` assembles `jcbtc/Qwen3.8-Flash-CIRU- +STRIX-Orca` (HuggingFace, public and ungated) into +`/home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca/`, sha256-verified. The +launcher refuses to start without the main GGUF, the MTP head and the PLE +sidecar: + +| file | bytes | role | +|---|---|---| +| `Qwen3.8-Flash-CIRU-STRIX-Orca.gguf` | 79,397,818,912 | the main model (Orca variant) | +| `mtp/Qwen3.8-Flash-CIRU-STRIX-Orca-MTP-Q8_0.gguf` | 4,135,893,440 | the MTP drafter head | +| `ple/ple.payload.bin` | 52,429,053,952 | the per-layer-embedding (PLE) payload — mmap'd and paged, not resident | +| `ple/ple.manifest.json` | — | PLE manifest | +| `ple/ple.scale.bf16` | — | PLE scale | +| `vision/mmproj-Qwen3.8-Flash-Orca-F16.mmproj` | 904,004,320 | the F16 vision projector (on disk, `ENABLE_VISION=0`) | + +The main GGUF ships as **transport parts** that the download script assembles +and verifies; the PLE is the large one (52.4 GB) and is paged by the engine's +own workers, not held in RAM. + +## Deployed config + +Tunables live in `scripts/frank-large.env`; the launcher owns the flag list +(`-sm none` one iGPU no split, `-ngl all`, f16 KV, flash attention, +cont-batching, the PLE sidecar, the MTP drafter, the production kernel-bundle +profile). The departures worth naming: + +| flag | value | why | +|---|---|---| +| `CONTEXT_SIZE` | `262144` | the model's FULL trained window (the operator's "full context" call); the GGUF carries no rope scaling | +| `PARALLEL_SLOTS` | `1` | one stream — and the MTP drafter **requires exactly one slot** in this runtime, so the operator's ask and the drafter's constraint are the same number | +| `ENABLE_MTP` | `1` | the MTP drafter, depth 4 (the Orca variant's own `mtp_depth_default`) | +| `ENABLE_VISION` | `0` | text-only for now; the F16 projector is on disk, one flip to enable. The fleet's image path is [[frank-small-gemma4-12b]] either way | +| `PLE_CACHE_MIB` | `4096` | the PLE pager's hot window kept resident; the vendor production-profile default. The 52.4 GB payload is mmap'd and paged around it | +| `HOST` | `0.0.0.0` | the launcher consumes it for `--host`; the value doubles as a JIT-cache directory name, which is why `0.0.0.0` is safe (HIP reads `HOST` bare — see the AGENTS.md note) | + +`TEMP`, `HOST` and `GPU_MAX_HEAP_SIZE` are the three names to avoid for any +new tunable: HIP reads them bare. + +## MTP + +The Multi-Token Prediction drafter is a **separate Q8_0 head** (not inside the +main GGUF), loaded by the launcher. It requires exactly one slot, which is why +`PARALLEL_SLOTS=1`. Depth 4 is the Orca variant's default. + +Verified at deploy (2026-09-19): the per-task `draft acceptance = 0.65625 +(21/32), mean len = 3.62` — the drafter is engaged and accepting, not silently +standing down. + +## PLE + +The per-layer-embedding payload (52.4 GB) is the Qwen3.8-Flash-Next +architecture's signature: token embeddings live outside the main weight set and +are paged on demand by the engine's own workers (`GGML_QWEN4EXP_PLE_WORKERS=16`, +`GGML_QWEN4EXP_PLE_IO_WORKERS=32` in the production profile). `PLE_CACHE_MIB=4096` +is the hot window kept resident; the rest is mmap'd file cache. **Do not treat +the 52.4 GB as resident memory** — `free` and the dashboards will not show it +pinned, and that is correct. + +## Client contract + +The differences from a stock llama.cpp endpoint that bite: + +- **`reasoning_content`** carries the reasoning trace (the standard OpenAI + field). The serve script passes `--chat-template-kwargs + '{"preserve_thinking":true}'` so prior thinking blocks are kept across turns; + a request-level `chat_template_kwargs` still wins. +- **`reasoning_effort` is gated.** Qwen3.8's template accepts only + `xhigh`, `medium`, `low` and **500s on anything else** (not a clamp). The + harness folds every level onto that set in `models.yml` + (`reasoningEffortMap`: minimal/low→low, medium→medium, high/max→xhigh) — + subagents run at `high`, which is why the map exists. +- **Text only.** `ENABLE_VISION=0`, so image content parts are refused. The + fleet's image path is [[frank-small-gemma4-12b]]. +- **Sampling.** The server has no sampler defaults of its own; the harness + applies the per-model recipe from `models.yml` (temp 1.0 / top_p 0.95 / top_k + 20 / min_p 0.0). A client that stays silent gets the launcher's default. +- **`tools` works** — the launcher passes `--jinja`, and the Qwen3.8 template + supplies the tool-call grammar. Verified at deploy: a tool call returns + `tool_calls`. + +## Deploy and operate + +```bash +./scripts/deploy.sh dankclaw --service frank-large +``` + +The handler pulls the checkout (ff-only), (re)links the unit, and gates on +`/health`. First boot is the slow one (model load ~46 s + graph capture); a +warm restart is fast. + +```bash +sudo systemctl status frank-large # the root unit +sudo journalctl -u frank-large -n 60 # startup account: runtime check, model load, "listening on http://0.0.0.0:7770", MTP draft loaded +curl -s http://127.0.0.1:7770/health # ok +curl -s http://127.0.0.1:7770/v1/models # Qwen3.8-Flash-CIRU-STRIX-Orca +``` + +**Stop and mask before any GPU experiment** (the AGENTS.md bench protocol): +`sudo systemctl stop frank-large && sudo systemctl mask frank-large`. The +frank-large serve process is a `llama-server` **and so is frank-small on the +same host** — never `pkill -x llama-server`. If you must kill by pattern, use +`pkill -f 'Qwen3.8-Flash-CIRU-STRIX-Orca'` (the model path is unique to this +endpoint and does not appear in the deploying shell's own command line). + +## Troubleshooting + +| symptom | cause | +|---|---| +| launcher exits 2: "The v4.4 HIP/ROCr runtime is missing" | `runtime/{hip,rocr}/lib/*.so` absent — the serve script's `ln -sfn` did not run, or the venv was moved. Re-run the serve script or rebuild: `bash $RUNTIME_DIR/scripts/ciru/setup-linux-amd.sh --install-host-deps` | +| "missing SDK venv runtime under .../.venv-rocm" | the venv's unversioned devel symlinks are gone — rebuild the runtime | +| "missing target model / missing payload" | a download did not finish — re-run `scripts/download-frank-large-orca.sh` | +| loads, `/health` ok, first completion slow (~45-60 s) | the PLE pager pulling the hot window in from file cache — not a crash, do not restart over it | +| a subagent 500s naming `reasoning_effort` | the Qwen3.8 template gate — the harness `reasoningEffortMap` must fold the level onto {xhigh, medium, low} | +| `free` / dashboards do not show 52 GB pinned | correct — the PLE is mmap'd file cache, not resident; only `PLE_CACHE_MIB` is hot | + +Retrieval: this is a **root** unit, so `journalctl -u frank-large` is the +durable record (no `--user`, no `sudo` needed for the journal once the unit is +readable). There is no container log — there is no container. + +## Rollback + +The predecessor is intact: the halogen payload stays at +`/home/dank/models/halogen` (retired, not deleted), and the halogen quadlet is +archived under `.AGENTTRASH/`. Rolling back is a `git revert` of this change set +plus a deploy — **but the root unit and the halogen quadlet both claim port +7770**, so the handler's port-free gate covers that race. See +[[halogen-flash-server]]. + +The `amd_iommu=off` kernel argument is separate and reverts on its own — see +[[dankclaw-iommu-translation-off]]. + +## Citations + +- [jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca](https://huggingface.co/jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca) — weights, MTP head, PLE payload, and the v4.4.1 QSA window-hotfix source tarball +- [CIRU runtime v4.4.1](https://github.com/ciru) — `scripts/ciru/run-server.sh` (runtime gate), `profiles/strix-halo-production.env` (kernel bundle), `docs/BUILD_LINUX.md` diff --git a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md index 72fb5ee..7a7c8b7 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md +++ b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md @@ -1,7 +1,7 @@ --- type: configuration -title: frank-medium llama.cpp Runbook (MTP drafter) -description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the model's own MTP head (the z-lab DFlash2 sidecar kept on disk, one env change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a 35:30 split (skewed toward the x16-slot card; see the 2026-09-17 part 3 section), a 393216-position KV pool (196608 per conversation over two slots), vision via the BF16 mmproj. +title: frank-medium llama.cpp Runbook (DFlash2 drafter) +description: The ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on Dankserver serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a balanced 32:33 split, a 327680-position KV pool (163840 per conversation over two slots), f16 KV plus a 16 GiB host-RAM prompt cache, vision via the BF16 mmproj. tags: - frank-medium - dankserver @@ -12,16 +12,19 @@ tags: - gfx1201 - r9700 - layer-split -timestamp: 2026-09-17 +timestamp: 2026-09-19 --- # frank-medium — llama.cpp (DFlash2 drafter) -The ACTIVE frank-medium deployment since 2026-09-16: **mainline llama.cpp** -serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at `UD-Q8_K_XL` -with the **z-lab DFlash2** block-diffusion drafter, as **one process across both -Radeon AI PRO R9700 cards**, layer-split. DSpark was tried in the drafter's -place for a few hours the same day and reverted — see *The DSpark excursion*. +The ACTIVE frank-medium deployment since 2026-09-16 (model swapped to +orcarouter Q8_0 on 2026-09-19): **mainline llama.cpp** +serving `orcarouter/Qwen3.8-27B-Uncensored-GGUF` at `Q8_0` +with the **z-lab DFlash2** `Q4_K_M` block-diffusion drafter, as **one process across both +Radeon AI PRO R9700 cards**, layer-split at a balanced 32:33. The predecessor +model (ajgazin Swift-Qwen3.8-27B UD-Q8_K_XL, 35:30 skew, 393216 pool) is the +calibration history in the sizing tables below. DSpark was tried in the drafter's +place on 2026-09-16 and reverted — see *The DSpark excursion*. It replaced the vllm-radiance MXFP4 replica pair + routing proxy ([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in the same day the pair went live. It is also the box's **only** LLM runtime: the diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index d35dec6..0f0e7ef 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -1,13 +1,13 @@ # LLM Fleet — Knowledge Base -Documentation for the LLM inference fleet on the two hosts (2026-09-16): +Documentation for the LLM inference fleet on the two hosts (2026-09-19): **mainline llama.cpp on Dankserver** serving frank-medium as one layer-split -process across both Radeon AI PRO R9700 cards (ajgazin Swift-Qwen3.8-27B- -Uncensored-Dynamic-MTP UD-Q8_K_XL + the z-lab DFlash2 drafter), and TWO -endpoints on Dankclaw — halogen-flash-server (frank-large, Qwen3.8-Flash-Next -w4b on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, -ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with MTP and vision on the RTX 3090, port -7771). +process across both Radeon AI PRO R9700 cards (orcarouter Qwen3.8-27B- +Uncensored Q8_0 + the z-lab DFlash2 Q4_K_M drafter, balanced 32:33), and TWO +endpoints on Dankclaw — the CIRU v4.4.1 runtime (frank-large, jcbtc +Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU, port 7770) and mainline +llama.cpp (frank-small, HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on +the RTX 3090, port 7771). > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -23,22 +23,22 @@ graph LR gfx1201, 32 GiB each] B2[llama.cpp, layer split frank-medium, port 7770] - C[Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL -ONE process on both cards, 262144 ctx / 1 slot, DFlash2] + C[orcarouter Qwen3.8-27B Q8_0 + DFlash2 Q4_K_M +ONE process on both cards, 2x163840 ctx, balanced 32:33] A --> B2 --> C end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] E2[RTX 3090 (PCIe x4 dock) + gfx1151 iGPU 64 GiB unified memory] - E[halogen-flash-server 0.5.8 + E[CIRU v4.4.1 runtime frank-large, port 7770] - F[peonist-ai Qwen3.8-Flash-Next w4b -2 x 200192 ctx over one 400384-position pool] + F[jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b +262144 ctx / 1 slot, MTP depth 4 + PLE] G[mainline llama.cpp 37b53fd frank-small, port 7771] - H[Qwen3.8-27B GSQ-RCO IQ3_S -131072 ctx / 1 slot, MTP + vision] + H[Gemma4-12B QAT Q4_K_M +4 x 262144 ctx, MTP + vision + audio] E2 --> E --> F E2 --> G --> H end @@ -57,11 +57,12 @@ frank-small, port 7771] ## Active Deployment - [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since - 2026-09-16: mainline llama.cpp serving - ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF at UD-Q8_K_XL with the - z-lab DFlash2 Q8_0 drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE - layer-split process behind :7770 (tensor parallelism is impossible on this - host — the second card cannot dispatch an RCCL collective) + 2026-09-16 (model swapped 2026-09-19): mainline llama.cpp serving + orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M + drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE layer-split process + behind :7770 (tensor parallelism is impossible on this host — the second card + cannot dispatch an RCCL collective), balanced 32:33 over two 163,840-token + slots - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16, one day after it was deployed: the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 + ggz14/radiance-vllm-mxfp4 patches) serving @@ -81,12 +82,17 @@ frank-small, port 7771] (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[halogen-flash-server]] — ACTIVE frank-large runbook again since 2026-09-16 - (rootless Quadlet, port 7770): peonist-ai halogen-flash-server 0.5.8 serving - peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn + quality - overlay) on the gfx1151 iGPU — two 200k conversations over one +- [[frank-large-ciru-orca]] — ACTIVE frank-large runbook since 2026-09-19 + (root systemd unit, port 7770): the CIRU v4.4.1 runtime (private host build, + QSA hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 + iGPU — one 262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, + runtime-root gate, client contract, troubleshooting +- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by + [[frank-large-ciru-orca]]; rollback target): peonist-ai halogen-flash-server + 0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b + .hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one 400,384-position pool, deployed flags, memory model, client contract, - troubleshooting + troubleshooting (2026-09-16 → 09-19) - [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17 (root systemd unit, port 7771): mainline llama.cpp serving HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX @@ -140,8 +146,8 @@ frank-small, port 7771] - [[kairic-edge]] — the retired Kairic Edge IU4 experiment (deployed and reverted 2026-08-22) - [[Dankserver System Specs]] — the 2x R9700 box (frank-medium) -- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on - halogen-flash-server since 2026-09-16, frank-small on the RTX 3090) +- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on the + CIRU v4.4.1 runtime since 2026-09-19, frank-small on the RTX 3090) - [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as the retirement record — the IOMMU/SVA err=-19 root cause and where diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 15dfc34..8e99a3c 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -116,12 +116,12 @@ The LLM inference fleet on the two hosts. Full index: - [[deployment]] — unified deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change - [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since - 2026-09-16: mainline llama.cpp (gfx1201 HIP build) serving - ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF at UD-Q8_K_XL with - the DFlash2 drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF at Q8_0) on - Dankserver's 2x Radeon AI PRO R9700 as one layer-split process behind the - :7770 endpoint — no tensor parallelism, because the second card cannot - dispatch an RCCL collective + 2026-09-16 (model swapped 2026-09-19): mainline llama.cpp (gfx1201 HIP build) + serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the DFlash2 + drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF at Q4_K_M) on Dankserver's 2x Radeon + AI PRO R9700 as one layer-split process behind the :7770 endpoint, balanced + 32:33 over two 163,840-token slots — no tensor parallelism, because the + second card cannot dispatch an RCCL collective - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16: the vllm-radiance MXFP4 stack serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on Dankserver's 2x Radeon AI PRO R9700 as two single-GPU replicas behind a @@ -145,11 +145,17 @@ The LLM inference fleet on the two hosts. Full index: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB, Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30; rollback record) -- [[halogen-flash-server]] — the ACTIVE frank-large runbook again since - 2026-09-16 (rootless Quadlet, port 7770): peonist-ai/halogen-flash-server - 0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (w4b `.hgn` + quality - overlay) on the gfx1151 iGPU — two 200k conversations over one - 400,384-position pool, deployed flags, memory model, client contract +- [[frank-large-ciru-orca]] — the ACTIVE frank-large runbook since 2026-09-19 + (root systemd unit, port 7770): the CIRU v4.4.1 runtime (private host build, + QSA hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 + iGPU — one 262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, + runtime-root gate, client contract +- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by + [[frank-large-ciru-orca]]; rollback target, 2026-09-16 → 09-19): + peonist-ai/halogen-flash-server 0.5.8 serving peonist-ai/halogen-qwen3.8- + flash-next (w4b `.hgn` + quality overlay) on the gfx1151 iGPU — two 200k + conversations over one 400,384-position pool, deployed flags, memory model, + client contract - [[frank-large-ik-uncensored]] — the RETIRED 2026-09-16 frank-large runbook and rollback target (2026-09-15 → 09-16): ik_llama.cpp serving mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 +