dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: wrap up the 2026-09-19 fleet cutover

Browse Source 
frank-large -> CIRU v4.4.1 Orca (new runbook), frank-medium -> the deployed
orcarouter Q8_0 / DFlash2 Q4_K_M / balanced 32:33 / 327680 state. The d374ab9
frank-medium deploy updated only scripts/tests and left AGENTS.md, the runbook
and both indexes on the predecessor (ajgazin Swift UD-Q8_K_XL, 35:30, 393216).

- AGENTS.md: frank-large section (CIRU Orca, root unit, runtime gate, PLE, MTP)
  + frank-medium section and topology row to the deployed shape
- new runbook frank-large-ciru-orca.md
- frank-medium runbook header/intro: orcarouter Q8_0 swap, 32:33, 327680;
  predecessor kept as calibration history
- LLM index.md: intro + mermaid (halogen->CIRU, GSQ-RCO->Gemma4, Swift->orcarouter),
  add ciru-orca, retire halogen
- top-level index.md: add ciru-orca link (docs-index seam), retire halogen,
  fix frank-medium entry
This commit is contained in:
Frank
2026-09-19 06:12:33 -07:00
parent f68bda56c7
commit 764cf41222
5 changed files with 340 additions and 106 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+61 -61
View File
@@ -12,7 +12,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service. Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service.
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen-flash-server on Alloy, Pi-hole) plus TWO LLM services: `frank-large` (CIRU v4.4.1 llama.cpp on
the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
Dankclaw reverse-proxies both hosts. Dankclaw reverse-proxies both hosts.
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
@@ -24,8 +24,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port | | Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------| |--------|---------|---------|-------|------|
|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (35:30 — skewed toward the x16-slot card; the other sits behind a Gen4 x1 chipset path), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj)|7770| |Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|orcarouter Qwen3.8-27B-Uncensored Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), 2x Radeon AI PRO R9700 in ONE layer-split process (32:33 balanced; ROCm0 Gen4 x16, ROCm1 Gen4 x1 chipset path), 327680 ctx over 2 slots of 163840, batch 6144x512, f16 KV + 16 GiB host-RAM prompt cache, vision (BF16 mmproj)|7770|
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 | | Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 |
Retired endpoints (`frank-npu`, and every prior runtime) are in Retired endpoints (`frank-npu`, and every prior runtime) are in
@@ -68,8 +68,8 @@ All edits flow through git. Never edit files on a remote machine directly.
The planner maps changed paths to services; the remote runner restarts only The planner maps changed paths to services; the remote runner restarts only
the selected service and records its revision outside the checkout. the selected service and records its revision outside the checkout.
Documentation-only changes are a no-op. LLM services are selected only by Documentation-only changes are a no-op. LLM services are selected only by
their own files — frank-large by `dankclaw/frank-large/quadlet/`, frank-small their own files — all three by their unit/env/serve trio in `scripts/` — so
and frank-medium by their unit/env/serve trio in `scripts/` — so unrelated unrelated
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
runs on a host without passwordless sudo; the value never enters the repo. runs on a host without passwordless sudo; the value never enters the repo.
@@ -207,9 +207,8 @@ archive.
**STOP the production service BEFORE benchmarking** — these servers serve **STOP the production service BEFORE benchmarking** — these servers serve
real users, and a bench beside a live service starves the GPU and corrupts real users, and a bench beside a live service starves the GPU and corrupts
results. Use the runtime's own harness: `llama-bench` (llama.cpp), `vllm results. Use the runtime's own harness: `llama-bench` for every llama.cpp
bench` (vLLM), and halogen-flash-server's own `bench`/`sweep` modes for endpoint (all three are llama.cpp now). Rolling your own bench
frank-large, which is the active runtime there again. Rolling your own bench
requires asking first. requires asking first.
Scratch ports are not used on LLM hosts — a Scratch ports are not used on LLM hosts — a
test process cannot share the single GPU anyway; validate against the service test process cannot share the single GPU anyway; validate against the service
@@ -227,11 +226,8 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
merely wedged kernel usually still answers ARP. merely wedged kernel usually still answers ARP.
1. Stop and mask (self-healing would restart it): 1. Stop and mask (self-healing would restart it):
- Dankclaw `frank-large` (rootless Quadlet again since 2026-09-16): - Dankclaw `frank-large` (root unit, CIRU llama.cpp on the iGPU):
`systemctl --user stop frank-large`, then move its `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
`~/.config/containers/systemd/frank-large.container` aside and
`systemctl --user daemon-reload` — the quadlet generator re-creates that
unit from the file on every reload, so the source is the lever, not `mask`.
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090): - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
`sudo systemctl stop frank-small && sudo systemctl mask frank-small` `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
- Dankserver `frank-medium` (system unit): - Dankserver `frank-medium` (system unit):
@@ -264,42 +260,45 @@ provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
`models.yml`, then fix every `modelRoles` entry that named the old endpoint — `models.yml`, then fix every `modelRoles` entry that named the old endpoint —
a role pointing at a provider the file does not define fails at call time. As a role pointing at a provider the file does not define fails at call time. As
of 2026-09-16 the image path is `frank-small` (`input: [text, image]`), and of 2026-09-16 the image path is `frank-small` (`input: [text, image]`), and
`frank-large` is text-only because its halogen deployment loads no vision `frank-large` is text-only because its CIRU deployment loads no vision
tower. There is **no audio** input anywhere in the fleet: that capability left projector (the F16 mmproj is on disk, one env flip away). There is **no audio**
with the FLM NPU endpoint on 2026-09-11. input anywhere in the fleet: that capability left with the FLM NPU endpoint on
2026-09-11.
### frank-large — halogen-flash-server (Dankclaw, ACTIVE) ### frank-large — CIRU v4.4.1 Orca (Dankclaw, ACTIVE)
halogen-flash-server 0.5.8 as a **rootless Quadlet again since 2026-09-16** CIRU v4.4.1 (the QSA window hotfix) as a **root systemd unit** (`scripts/
(`dankclaw/frank-large/quadlet/frank-large.container`), serving frank-large.service`, `User=dank`), serving `jcbtc/Qwen3.8-Flash-CIRU-STRIX-
`peonist-ai/halogen-qwen3.8-flash-next` (115.55 GiB w4b `.hgn` checkpoint + a Orca` (79.4 GiB main GGUF + a 4.1 GiB MTP head + a 52.4 GiB PLE payload, all
quality overlay that re-fetched to 2.40 GiB on 2026-09-16) on port 7770 from sha256-verified) on port 7770 from the gfx1151 iGPU. One stream, the model's
the gfx1151 iGPU. Two conversations of 200,192 tokens resident (pool 400,384, FULL 262,144 trained window (`PARALLEL_SLOTS=1`; the MTP drafter needs exactly
2 slots), greedy default, reasoning extracted to `reasoning_content`, and **no one slot in this runtime), MTP depth 4, PLE cache 4096 MiB, reasoning extracted
vision tower** — that moved to frank-small. It replaced ik_llama.cpp, which had to `reasoning_content`, and **no vision projector** (the F16 mmproj is on disk,
replaced it for two days. `ENABLE_VISION=0` — one env flip to enable). It replaced halogen-flash-server,
which had replaced ik_llama.cpp.
The engine **fetches its own weights on first start** (`HALOGEN_DOWNLOAD`, into The runtime is a **private host build** (`/home/dank/GitPackages/ciru-runtime-
the `/home/dank/models/halogen` bind mount): clearing that directory and v4.4.1-qsa-hotfix/`), not a container image: a gfx1151 HIP build of the CIRU
restarting the unit is the supported way to pick up upstream updates, and it llama.cpp fork with its own bundled ROCm SDK in a venv (`.venv-rocm/`), and the
needs the directory to exist **empty** — a missing directory fails the binary resolves the SDK libs via RUNPATH. The launcher (`scripts/ciru/
container with `Error: statfs /home/dank/models/halogen: no such file or run-orca-server.sh` → `run-server.sh`) has a **runtime gate**: it exits 2 unless
directory` and the unit loops on `Restart=on-failure` until `mkdir -p` brings `${repo_root}/runtime/hip/lib/libamdhip64.so` and `runtime/rocr/lib/
it back. The sidecars are versioned in place, so a stale copy is invisible from libhsa-runtime64.so` exist. The serve script (`scripts/serve-frank-large.sh`)
the file name (both overlays were ~95 MB larger upstream than the copies this satisfies that gate idempotently by `ln -sfn`-ing both to the venv's unversioned
host had been serving). devel symlinks, then `exec`s the launcher with `--chat-template-kwargs
'{"preserve_thinking":true}'` (Qwen3.8's template needs this to carry thinking
blocks across turns; a request-level value still wins).
Measured boot ledger (2026-09-16, `HALOGEN_VERBOSE=1` ships in the unit): The serve script validates launcher → built binary → venv runtime → main GGUF →
device budget `121 − 67.7 − 20 = 33.5 GiB` against a need of `~27.2 GiB`, so MTP head → PLE payload + manifest + scale, and exits with a pointer message if
the configured pool was **not clamped**; 68.0 GiB of weights locked + 11.0 GiB any is missing. `PLE_CACHE_MIB=4096` is the vendor production-profile default;
of KV pool + 16.5 GiB of working memory = 95.4 GiB in all, leaving 14.7 GiB the 52.4 GiB PLE is mmap'd and paged, not resident.
(475 contiguous 2 MiB blocks) for everything else: frank-small on the 3090 and
~28 co-tenant containers. The engine printed its co-tenant stall NOTE and
logged 4,350 compaction stalls (285 failed) while reserving the pool, so read
that as "tight and fragmented", not "out of memory".
Runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`. The halogen Quadlet (`dankclaw/frank-large/quadlet/`) was retired to
Rollback: [[frank-large-ik-uncensored]]. `.AGENTTRASH/` — one owner per endpoint, now the scripts/ trio.
Runbook: `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md`.
Rollback: [[halogen-flash-server]].
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
@@ -327,25 +326,25 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`. Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
Rollback: [[frank-small-qwen38-gsq-rco]]. Rollback: [[frank-small-qwen38-gsq-rco]].
### frank-medium — llama.cpp layer split, MTP drafter (Dankserver, ACTIVE) ### frank-medium — llama.cpp layer split, DFlash2 drafter (Dankserver, ACTIVE)
Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root
systemd unit `scripts/frank-medium.service`) serving systemd unit `scripts/frank-medium.service`) serving
`ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL** `orcarouter/Qwen3.8-27B-Uncensored-GGUF` at **Q8_0** (27.0 GiB) with the
(29.29 GiB) with the model's OWN **MTP head** (`blk.<n>.nextn.*` in the file; **z-lab DFlash2 `Q4_K_M` drafter** (1.06 GiB). The target file also carries the
the z-lab DFlash2 `Q8_0` sidecar stays on disk and in the fetch script — one model's own MTP head (`blk.<n>.nextn.*`, `nextn_predict_layers=1`), so
`SPEC_TYPE` change brings the faster chain back), in **one process across both `draft-mtp` is one `SPEC_TYPE` change away — the drafter file stays on disk. In
R9700s**, `--split-mode layer` at a **35:30 tensor split** — one layer moved **one process across both R9700s**, `--split-mode layer` at a **32:33 tensor
to ROCm0 on 2026-09-17 because the cards' PCIe paths are not symmetric: split** — balanced (operator call 2026-09-19, the most even split 65 layers
ROCm0 (03:00.0) is CPU-root-port Gen4 x16 (~31.5 GB/s) while ROCm1 allow; ROCm1, the last-layers card that also carries the drafter and output
(0b:00.0) rides the B650 chipset at Gen4 x1 (~1.97 GB/s), and with the layers, holds the extra one). The predecessor's 35:30 skew (2026-09-17) leaned
drafter, output layers and heavier compute buffers pinned to the work onto the x16 card because ROCm1 (0b:00.0) rides the B650 chipset at Gen4
last-layers card, the bottlenecked card was running 98% while the x16 card x1 (~1.97 GB/s) while ROCm0 (03:00.0) is CPU-root-port Gen4 x16 (~31.5 GB/s);
sat at 40%. 36:29 would cross a full-attention layer and fails both the the balanced point trades that lean for symmetry and was accepted. 327680 ctx
1 GiB floor and the fit slack, so 35:30 is the farthest safe shift at this over **2 slots** of 163840 — inside the checkpoint's trained 262144 window, so
pool. 393216 ctx over **2 slots** of 196608, batch 6144x512, vision no rope extrapolation (unlike the predecessor's 393216) — batch 6144x512, f16
via the BF16 mmproj. DSpark was KV, a 16384 MiB host-RAM prompt cache, vision via the BF16 mmproj. DSpark was
put up in the drafter's place the same day and reverted: it declares no sliding tried in the drafter's place on 2026-09-16 and reverted: it declares no sliding
window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four
layers of the split. layers of the split.
@@ -706,7 +705,7 @@ Active:
| Service | Runbook | | Service | Runbook |
|---------|---------| |---------|---------|
| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` | | frank-large (CIRU v4.4.1 Orca, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` |
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` | | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
| frank-medium (llama.cpp + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` | | frank-medium (llama.cpp + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
@@ -718,6 +717,7 @@ Retired (rollback targets and history):
|---------|------|--------|---------| |---------|------|--------|---------|
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` | | kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` | | ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` | | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` | | vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` | | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
serverAppConfigs-docs/LLM/frank-large-ciru-orca.md
+219
View File
@@ -0,0 +1,219 @@
---
type: reference
title: CIRU v4.4.1 Orca on Dankclaw (frank-large)
description: The active frank-large runbook — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing.
tags:
- llm
- ciru
- dankclaw
- frank-large
- strix-halo
- gfx1151
- qwen4exp
- rocm
- mtp
timestamp: 2026-09-19
---
# CIRU v4.4.1 Orca on Dankclaw (frank-large)
> **ACTIVE since 2026-09-19.** It replaced halogen-flash-server (see
> [[halogen-flash-server]], the current rollback), which had replaced
> ik_llama.cpp (see [[frank-large-ik-uncensored]]). Deployed as a **root
> systemd unit** (`scripts/frank-large.service`, `User=dank`), port 7770, on the
> same gfx1151 iGPU the halogen quadlet used. The RTX 3090 it does not touch
> belongs to [[frank-small-gemma4-12b]].
CIRU is a llama.cpp fork tuned for the Qwen3.8-Flash-Next ("qwen4exp")
architecture on AMD gfx1151 (Strix Halo). v4.4.1 carries a **QSA window
hotfix** that is what makes vision and the MTP drafter work on this build; the
pre-applied hotfix source ships in the model repo under
`runtime/v4.4.1/qsa-window-hotfix/`. Like every other fleet endpoint it is a
llama.cpp runtime — `llama-bench` is its own harness.
## Shape
One root unit, one process, one published port:
| half | port | reachable |
|---|---|---|
| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
Unlike the halogen predecessor (two ports: an unauthenticated engine on 8730
and an OpenAI front-end on 8731), CIRU is a single `llama-server` process —
there is no separate engine port to keep private.
`/v1/models` reports the loaded model name, **`Qwen3.8-Flash-CIRU-STRIX-Orca`**
(the launcher sets no `--alias`). The endpoint serves **whatever model name a
request names** — llama.cpp does not reject an unknown id — so the harness's
`frank-large` label in `models.yml` works without a rename.
## Runtime
A **private host build**, not a container image. The v4.4.1 source with the QSA
window hotfix, built on the host against the private ROCm SDK that
`scripts/ciru/setup-linux-amd.sh` installs into the checkout's own venv:
| path | what |
|---|---|
| `/home/dank/GitPackages/ciru-runtime-v4.4.1-qsa-hotfix/` | the checkout (source + build + venv) |
| `build-gfx1151-sdk/bin/llama-server` | the built binary |
| `.venv-rocm/` | the private ROCm SDK venv (~5.7 GB); the binary resolves its libs through RUNPATH |
| `scripts/ciru/run-orca-server.sh` | the launcher (`exec`'d by the serve script) |
| `profiles/strix-halo-production.env` | the 36-variable kernel bundle + PLE pager defaults, sourced by the launcher AFTER `frank-large.env` (every line is `${VAR:-default}`, so a value set in `frank-large.env` wins) |
### The runtime gate
`run-orca-server.sh` sources `run-server.sh`, which **exits 2** unless
`${repo_root}/runtime/hip/lib/libamdhip64.so` and `runtime/rocr/lib/
libhsa-runtime64.so` exist (unversioned, no suffix). It prepends both to
`LD_LIBRARY_PATH` and `ldd`-checks that the built `libllama-common.so` carries
the "Qwen4Exp MTP image position" marker and the HIP lib carries the
`flash_attn_index_mask_clear/set/empty` symbols before exec.
The build links the venv through RUNPATH, so those two files are an existence
gate, not a real dependency. `serve-frank-large.sh` satisfies the gate
idempotently: it finds the venv's unversioned devel symlinks
(`.venv-rocm/lib/python*/site-packages/_rocm_sdk_devel/lib/`) and `ln -sfn`-s
both into `runtime/{hip,rocr}/lib/`. **The first deploy of this endpoint failed
exactly here** — the build linked the venv but `runtime/` did not exist, so the
launcher exited 2 and the unit crash-looped until the serve script created the
symlinks.
## Payload
`scripts/download-frank-large-orca.sh` assembles `jcbtc/Qwen3.8-Flash-CIRU-
STRIX-Orca` (HuggingFace, public and ungated) into
`/home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca/`, sha256-verified. The
launcher refuses to start without the main GGUF, the MTP head and the PLE
sidecar:
| file | bytes | role |
|---|---|---|
| `Qwen3.8-Flash-CIRU-STRIX-Orca.gguf` | 79,397,818,912 | the main model (Orca variant) |
| `mtp/Qwen3.8-Flash-CIRU-STRIX-Orca-MTP-Q8_0.gguf` | 4,135,893,440 | the MTP drafter head |
| `ple/ple.payload.bin` | 52,429,053,952 | the per-layer-embedding (PLE) payload — mmap'd and paged, not resident |
| `ple/ple.manifest.json` | — | PLE manifest |
| `ple/ple.scale.bf16` | — | PLE scale |
| `vision/mmproj-Qwen3.8-Flash-Orca-F16.mmproj` | 904,004,320 | the F16 vision projector (on disk, `ENABLE_VISION=0`) |
The main GGUF ships as **transport parts** that the download script assembles
and verifies; the PLE is the large one (52.4 GB) and is paged by the engine's
own workers, not held in RAM.
## Deployed config
Tunables live in `scripts/frank-large.env`; the launcher owns the flag list
(`-sm none` one iGPU no split, `-ngl all`, f16 KV, flash attention,
cont-batching, the PLE sidecar, the MTP drafter, the production kernel-bundle
profile). The departures worth naming:
| flag | value | why |
|---|---|---|
| `CONTEXT_SIZE` | `262144` | the model's FULL trained window (the operator's "full context" call); the GGUF carries no rope scaling |
| `PARALLEL_SLOTS` | `1` | one stream — and the MTP drafter **requires exactly one slot** in this runtime, so the operator's ask and the drafter's constraint are the same number |
| `ENABLE_MTP` | `1` | the MTP drafter, depth 4 (the Orca variant's own `mtp_depth_default`) |
| `ENABLE_VISION` | `0` | text-only for now; the F16 projector is on disk, one flip to enable. The fleet's image path is [[frank-small-gemma4-12b]] either way |
| `PLE_CACHE_MIB` | `4096` | the PLE pager's hot window kept resident; the vendor production-profile default. The 52.4 GB payload is mmap'd and paged around it |
| `HOST` | `0.0.0.0` | the launcher consumes it for `--host`; the value doubles as a JIT-cache directory name, which is why `0.0.0.0` is safe (HIP reads `HOST` bare — see the AGENTS.md note) |
`TEMP`, `HOST` and `GPU_MAX_HEAP_SIZE` are the three names to avoid for any
new tunable: HIP reads them bare.
## MTP
The Multi-Token Prediction drafter is a **separate Q8_0 head** (not inside the
main GGUF), loaded by the launcher. It requires exactly one slot, which is why
`PARALLEL_SLOTS=1`. Depth 4 is the Orca variant's default.
Verified at deploy (2026-09-19): the per-task `draft acceptance = 0.65625
(21/32), mean len = 3.62` — the drafter is engaged and accepting, not silently
standing down.
## PLE
The per-layer-embedding payload (52.4 GB) is the Qwen3.8-Flash-Next
architecture's signature: token embeddings live outside the main weight set and
are paged on demand by the engine's own workers (`GGML_QWEN4EXP_PLE_WORKERS=16`,
`GGML_QWEN4EXP_PLE_IO_WORKERS=32` in the production profile). `PLE_CACHE_MIB=4096`
is the hot window kept resident; the rest is mmap'd file cache. **Do not treat
the 52.4 GB as resident memory** — `free` and the dashboards will not show it
pinned, and that is correct.
## Client contract
The differences from a stock llama.cpp endpoint that bite:
- **`reasoning_content`** carries the reasoning trace (the standard OpenAI
field). The serve script passes `--chat-template-kwargs
'{"preserve_thinking":true}'` so prior thinking blocks are kept across turns;
a request-level `chat_template_kwargs` still wins.
- **`reasoning_effort` is gated.** Qwen3.8's template accepts only
`xhigh`, `medium`, `low` and **500s on anything else** (not a clamp). The
harness folds every level onto that set in `models.yml`
(`reasoningEffortMap`: minimal/low→low, medium→medium, high/max→xhigh) —
subagents run at `high`, which is why the map exists.
- **Text only.** `ENABLE_VISION=0`, so image content parts are refused. The
fleet's image path is [[frank-small-gemma4-12b]].
- **Sampling.** The server has no sampler defaults of its own; the harness
applies the per-model recipe from `models.yml` (temp 1.0 / top_p 0.95 / top_k
20 / min_p 0.0). A client that stays silent gets the launcher's default.
- **`tools` works** — the launcher passes `--jinja`, and the Qwen3.8 template
supplies the tool-call grammar. Verified at deploy: a tool call returns
`tool_calls`.
## Deploy and operate
```bash
./scripts/deploy.sh dankclaw --service frank-large
```
The handler pulls the checkout (ff-only), (re)links the unit, and gates on
`/health`. First boot is the slow one (model load ~46 s + graph capture); a
warm restart is fast.
```bash
sudo systemctl status frank-large # the root unit
sudo journalctl -u frank-large -n 60 # startup account: runtime check, model load, "listening on http://0.0.0.0:7770", MTP draft loaded
curl -s http://127.0.0.1:7770/health # ok
curl -s http://127.0.0.1:7770/v1/models # Qwen3.8-Flash-CIRU-STRIX-Orca
```
**Stop and mask before any GPU experiment** (the AGENTS.md bench protocol):
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`. The
frank-large serve process is a `llama-server` **and so is frank-small on the
same host** — never `pkill -x llama-server`. If you must kill by pattern, use
`pkill -f 'Qwen3.8-Flash-CIRU-STRIX-Orca'` (the model path is unique to this
endpoint and does not appear in the deploying shell's own command line).
## Troubleshooting
| symptom | cause |
|---|---|
| launcher exits 2: "The v4.4 HIP/ROCr runtime is missing" | `runtime/{hip,rocr}/lib/*.so` absent — the serve script's `ln -sfn` did not run, or the venv was moved. Re-run the serve script or rebuild: `bash $RUNTIME_DIR/scripts/ciru/setup-linux-amd.sh --install-host-deps` |
| "missing SDK venv runtime under .../.venv-rocm" | the venv's unversioned devel symlinks are gone — rebuild the runtime |
| "missing target model / missing payload" | a download did not finish — re-run `scripts/download-frank-large-orca.sh` |
| loads, `/health` ok, first completion slow (~45-60 s) | the PLE pager pulling the hot window in from file cache — not a crash, do not restart over it |
| a subagent 500s naming `reasoning_effort` | the Qwen3.8 template gate — the harness `reasoningEffortMap` must fold the level onto {xhigh, medium, low} |
| `free` / dashboards do not show 52 GB pinned | correct — the PLE is mmap'd file cache, not resident; only `PLE_CACHE_MIB` is hot |
Retrieval: this is a **root** unit, so `journalctl -u frank-large` is the
durable record (no `--user`, no `sudo` needed for the journal once the unit is
readable). There is no container log — there is no container.
## Rollback
The predecessor is intact: the halogen payload stays at
`/home/dank/models/halogen` (retired, not deleted), and the halogen quadlet is
archived under `.AGENTTRASH/`. Rolling back is a `git revert` of this change set
plus a deploy — **but the root unit and the halogen quadlet both claim port
7770**, so the handler's port-free gate covers that race. See
[[halogen-flash-server]].
The `amd_iommu=off` kernel argument is separate and reverts on its own — see
[[dankclaw-iommu-translation-off]].
## Citations
- [jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca](https://huggingface.co/jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca) — weights, MTP head, PLE payload, and the v4.4.1 QSA window-hotfix source tarball
- [CIRU runtime v4.4.1](https://github.com/ciru) — `scripts/ciru/run-server.sh` (runtime gate), `profiles/strix-halo-production.env` (kernel bundle), `docs/BUILD_LINUX.md`
serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md
+11 -8
View File
@@ -1,7 +1,7 @@
--- ---
type: configuration type: configuration
title: frank-medium llama.cpp Runbook (MTP drafter) title: frank-medium llama.cpp Runbook (DFlash2 drafter)
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the model's own MTP head (the z-lab DFlash2 sidecar kept on disk, one env change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a 35:30 split (skewed toward the x16-slot card; see the 2026-09-17 part 3 section), a 393216-position KV pool (196608 per conversation over two slots), vision via the BF16 mmproj. description: The ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on Dankserver serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a balanced 32:33 split, a 327680-position KV pool (163840 per conversation over two slots), f16 KV plus a 16 GiB host-RAM prompt cache, vision via the BF16 mmproj.
tags: tags:
- frank-medium - frank-medium
- dankserver - dankserver
@@ -12,16 +12,19 @@ tags:
- gfx1201 - gfx1201
- r9700 - r9700
- layer-split - layer-split
timestamp: 2026-09-17 timestamp: 2026-09-19
--- ---
# frank-medium — llama.cpp (DFlash2 drafter) # frank-medium — llama.cpp (DFlash2 drafter)
The ACTIVE frank-medium deployment since 2026-09-16: **mainline llama.cpp** The ACTIVE frank-medium deployment since 2026-09-16 (model swapped to
serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at `UD-Q8_K_XL` orcarouter Q8_0 on 2026-09-19): **mainline llama.cpp**
with the **z-lab DFlash2** block-diffusion drafter, as **one process across both serving `orcarouter/Qwen3.8-27B-Uncensored-GGUF` at `Q8_0`
Radeon AI PRO R9700 cards**, layer-split. DSpark was tried in the drafter's with the **z-lab DFlash2** `Q4_K_M` block-diffusion drafter, as **one process across both
place for a few hours the same day and reverted — see *The DSpark excursion*. Radeon AI PRO R9700 cards**, layer-split at a balanced 32:33. The predecessor
model (ajgazin Swift-Qwen3.8-27B UD-Q8_K_XL, 35:30 skew, 393216 pool) is the
calibration history in the sizing tables below. DSpark was tried in the drafter's
place on 2026-09-16 and reverted — see *The DSpark excursion*.
It replaced the vllm-radiance MXFP4 replica pair + routing proxy It replaced the vllm-radiance MXFP4 replica pair + routing proxy
([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in ([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in
the same day the pair went live. It is also the box's **only** LLM runtime: the the same day the pair went live. It is also the box's **only** LLM runtime: the
serverAppConfigs-docs/LLM/index.md
+32 -26
View File
@@ -1,13 +1,13 @@
# LLM Fleet — Knowledge Base # LLM Fleet — Knowledge Base
Documentation for the LLM inference fleet on the two hosts (2026-09-16): Documentation for the LLM inference fleet on the two hosts (2026-09-19):
**mainline llama.cpp on Dankserver** serving frank-medium as one layer-split **mainline llama.cpp on Dankserver** serving frank-medium as one layer-split
process across both Radeon AI PRO R9700 cards (ajgazin Swift-Qwen3.8-27B- process across both Radeon AI PRO R9700 cards (orcarouter Qwen3.8-27B-
Uncensored-Dynamic-MTP UD-Q8_K_XL + the z-lab DFlash2 drafter), and TWO Uncensored Q8_0 + the z-lab DFlash2 Q4_K_M drafter, balanced 32:33), and TWO
endpoints on Dankclaw — halogen-flash-server (frank-large, Qwen3.8-Flash-Next endpoints on Dankclaw — the CIRU v4.4.1 runtime (frank-large, jcbtc
w4b on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU, port 7770) and mainline
ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with MTP and vision on the RTX 3090, port llama.cpp (frank-small, HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on
7771). the RTX 3090, port 7771).
> 2026-08-22: the former `vllmScripts` repository is merged into the > 2026-08-22: the former `vllmScripts` repository is merged into the
> serverAppConfigs repository. This knowledge base moved from the old repo's > serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -23,22 +23,22 @@ graph LR
gfx1201, 32 GiB each] gfx1201, 32 GiB each]
B2[llama.cpp, layer split B2[llama.cpp, layer split
frank-medium, port 7770] frank-medium, port 7770]
C[Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL C[orcarouter Qwen3.8-27B Q8_0 + DFlash2 Q4_K_M
ONE process on both cards, 262144 ctx / 1 slot, DFlash2] ONE process on both cards, 2x163840 ctx, balanced 32:33]
A --> B2 --> C A --> B2 --> C
end end
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
E2[RTX 3090 (PCIe x4 dock) + E2[RTX 3090 (PCIe x4 dock) +
gfx1151 iGPU 64 GiB unified memory] gfx1151 iGPU 64 GiB unified memory]
E[halogen-flash-server 0.5.8 E[CIRU v4.4.1 runtime
frank-large, port 7770] frank-large, port 7770]
F[peonist-ai Qwen3.8-Flash-Next w4b F[jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b
2 x 200192 ctx over one 400384-position pool] 262144 ctx / 1 slot, MTP depth 4 + PLE]
G[mainline llama.cpp 37b53fd G[mainline llama.cpp 37b53fd
frank-small, port 7771] frank-small, port 7771]
H[Qwen3.8-27B GSQ-RCO IQ3_S H[Gemma4-12B QAT Q4_K_M
131072 ctx / 1 slot, MTP + vision] 4 x 262144 ctx, MTP + vision + audio]
E2 --> E --> F E2 --> E --> F
E2 --> G --> H E2 --> G --> H
end end
@@ -57,11 +57,12 @@ frank-small, port 7771]
## Active Deployment ## Active Deployment
- [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since - [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since
2026-09-16: mainline llama.cpp serving 2026-09-16 (model swapped 2026-09-19): mainline llama.cpp serving
ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF at UD-Q8_K_XL with the orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M
z-lab DFlash2 Q8_0 drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE layer-split process
layer-split process behind :7770 (tensor parallelism is impossible on this behind :7770 (tensor parallelism is impossible on this host — the second card
host — the second card cannot dispatch an RCCL collective) cannot dispatch an RCCL collective), balanced 32:33 over two 163,840-token
slots
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16, one day after it was - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16, one day after it was
deployed: the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 + deployed: the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 +
ggz14/radiance-vllm-mxfp4 patches) serving ggz14/radiance-vllm-mxfp4 patches) serving
@@ -81,12 +82,17 @@ frank-small, port 7771]
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
- [[deployment]] — deploy and update process for both servers - [[deployment]] — deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[halogen-flash-server]] — ACTIVE frank-large runbook again since 2026-09-16 - [[frank-large-ciru-orca]] — ACTIVE frank-large runbook since 2026-09-19
(rootless Quadlet, port 7770): peonist-ai halogen-flash-server 0.5.8 serving (root systemd unit, port 7770): the CIRU v4.4.1 runtime (private host build,
peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn + quality QSA hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151
overlay) on the gfx1151 iGPU — two 200k conversations over one iGPU — one 262,144-token conversation, MTP depth 4, 4096 MiB PLE cache,
runtime-root gate, client contract, troubleshooting
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
[[frank-large-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
400,384-position pool, deployed flags, memory model, client contract, 400,384-position pool, deployed flags, memory model, client contract,
troubleshooting troubleshooting (2026-09-16 → 09-19)
- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17 - [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
(root systemd unit, port 7771): mainline llama.cpp serving (root systemd unit, port 7771): mainline llama.cpp serving
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
@@ -140,8 +146,8 @@ frank-small, port 7771]
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment - [[kairic-edge]] — the retired Kairic Edge IU4 experiment
(deployed and reverted 2026-08-22) (deployed and reverted 2026-08-22)
- [[Dankserver System Specs]] — the 2x R9700 box (frank-medium) - [[Dankserver System Specs]] — the 2x R9700 box (frank-medium)
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on - [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on the
halogen-flash-server since 2026-09-16, frank-small on the RTX 3090) CIRU v4.4.1 runtime since 2026-09-19, frank-small on the RTX 3090)
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint - [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
the retirement record — the IOMMU/SVA err=-19 root cause and where the retirement record — the IOMMU/SVA err=-19 root cause and where
serverAppConfigs-docs/index.md
+17 -11
View File
@@ -116,12 +116,12 @@ The LLM inference fleet on the two hosts. Full index:
- [[deployment]] — unified deploy and update process for both servers - [[deployment]] — unified deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since - [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since
2026-09-16: mainline llama.cpp (gfx1201 HIP build) serving 2026-09-16 (model swapped 2026-09-19): mainline llama.cpp (gfx1201 HIP build)
ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF at UD-Q8_K_XL with serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the DFlash2
the DFlash2 drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF at Q8_0) on drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF at Q4_K_M) on Dankserver's 2x Radeon
Dankserver's 2x Radeon AI PRO R9700 as one layer-split process behind the AI PRO R9700 as one layer-split process behind the :7770 endpoint, balanced
:7770 endpoint — no tensor parallelism, because the second card cannot 32:33 over two 163,840-token slots — no tensor parallelism, because the
dispatch an RCCL collective second card cannot dispatch an RCCL collective
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16: the vllm-radiance - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16: the vllm-radiance
MXFP4 stack serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on MXFP4 stack serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on
Dankserver's 2x Radeon AI PRO R9700 as two single-GPU replicas behind a Dankserver's 2x Radeon AI PRO R9700 as two single-GPU replicas behind a
@@ -145,11 +145,17 @@ The LLM inference fleet on the two hosts. Full index:
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB, agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30; Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
rollback record) rollback record)
- [[halogen-flash-server]] — the ACTIVE frank-large runbook again since - [[frank-large-ciru-orca]] — the ACTIVE frank-large runbook since 2026-09-19
2026-09-16 (rootless Quadlet, port 7770): peonist-ai/halogen-flash-server (root systemd unit, port 7770): the CIRU v4.4.1 runtime (private host build,
0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (w4b `.hgn` + quality QSA hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151
overlay) on the gfx1151 iGPU — two 200k conversations over one iGPU — one 262,144-token conversation, MTP depth 4, 4096 MiB PLE cache,
400,384-position pool, deployed flags, memory model, client contract runtime-root gate, client contract
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
[[frank-large-ciru-orca]]; rollback target, 2026-09-16 → 09-19):
peonist-ai/halogen-flash-server 0.5.8 serving peonist-ai/halogen-qwen3.8-
flash-next (w4b `.hgn` + quality overlay) on the gfx1151 iGPU — two 200k
conversations over one 400,384-position pool, deployed flags, memory model,
client contract
- [[frank-large-ik-uncensored]] — the RETIRED 2026-09-16 frank-large runbook - [[frank-large-ik-uncensored]] — the RETIRED 2026-09-16 frank-large runbook
and rollback target (2026-09-15 → 09-16): ik_llama.cpp serving and rollback target (2026-09-15 → 09-16): ik_llama.cpp serving
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 + mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 +