docs: wrap up the 2026-09-19 fleet cutover

frank-large -> CIRU v4.4.1 Orca (new runbook), frank-medium -> the deployed
orcarouter Q8_0 / DFlash2 Q4_K_M / balanced 32:33 / 327680 state. The d374ab9
frank-medium deploy updated only scripts/tests and left AGENTS.md, the runbook
and both indexes on the predecessor (ajgazin Swift UD-Q8_K_XL, 35:30, 393216).

- AGENTS.md: frank-large section (CIRU Orca, root unit, runtime gate, PLE, MTP)
  + frank-medium section and topology row to the deployed shape
- new runbook frank-large-ciru-orca.md
- frank-medium runbook header/intro: orcarouter Q8_0 swap, 32:33, 327680;
  predecessor kept as calibration history
- LLM index.md: intro + mermaid (halogen->CIRU, GSQ-RCO->Gemma4, Swift->orcarouter),
  add ciru-orca, retire halogen
- top-level index.md: add ciru-orca link (docs-index seam), retire halogen,
  fix frank-medium entry
This commit is contained in:
Frank
2026-09-19 06:12:33 -07:00
parent f68bda56c7
commit 764cf41222
5 changed files with 340 additions and 106 deletions
@@ -0,0 +1,219 @@
---
type: reference
title: CIRU v4.4.1 Orca on Dankclaw (frank-large)
description: The active frank-large runbook — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing.
tags:
- llm
- ciru
- dankclaw
- frank-large
- strix-halo
- gfx1151
- qwen4exp
- rocm
- mtp
timestamp: 2026-09-19
---
# CIRU v4.4.1 Orca on Dankclaw (frank-large)
> **ACTIVE since 2026-09-19.** It replaced halogen-flash-server (see
> [[halogen-flash-server]], the current rollback), which had replaced
> ik_llama.cpp (see [[frank-large-ik-uncensored]]). Deployed as a **root
> systemd unit** (`scripts/frank-large.service`, `User=dank`), port 7770, on the
> same gfx1151 iGPU the halogen quadlet used. The RTX 3090 it does not touch
> belongs to [[frank-small-gemma4-12b]].
CIRU is a llama.cpp fork tuned for the Qwen3.8-Flash-Next ("qwen4exp")
architecture on AMD gfx1151 (Strix Halo). v4.4.1 carries a **QSA window
hotfix** that is what makes vision and the MTP drafter work on this build; the
pre-applied hotfix source ships in the model repo under
`runtime/v4.4.1/qsa-window-hotfix/`. Like every other fleet endpoint it is a
llama.cpp runtime — `llama-bench` is its own harness.
## Shape
One root unit, one process, one published port:
| half | port | reachable |
|---|---|---|
| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
Unlike the halogen predecessor (two ports: an unauthenticated engine on 8730
and an OpenAI front-end on 8731), CIRU is a single `llama-server` process —
there is no separate engine port to keep private.
`/v1/models` reports the loaded model name, **`Qwen3.8-Flash-CIRU-STRIX-Orca`**
(the launcher sets no `--alias`). The endpoint serves **whatever model name a
request names** — llama.cpp does not reject an unknown id — so the harness's
`frank-large` label in `models.yml` works without a rename.
## Runtime
A **private host build**, not a container image. The v4.4.1 source with the QSA
window hotfix, built on the host against the private ROCm SDK that
`scripts/ciru/setup-linux-amd.sh` installs into the checkout's own venv:
| path | what |
|---|---|
| `/home/dank/GitPackages/ciru-runtime-v4.4.1-qsa-hotfix/` | the checkout (source + build + venv) |
| `build-gfx1151-sdk/bin/llama-server` | the built binary |
| `.venv-rocm/` | the private ROCm SDK venv (~5.7 GB); the binary resolves its libs through RUNPATH |
| `scripts/ciru/run-orca-server.sh` | the launcher (`exec`'d by the serve script) |
| `profiles/strix-halo-production.env` | the 36-variable kernel bundle + PLE pager defaults, sourced by the launcher AFTER `frank-large.env` (every line is `${VAR:-default}`, so a value set in `frank-large.env` wins) |
### The runtime gate
`run-orca-server.sh` sources `run-server.sh`, which **exits 2** unless
`${repo_root}/runtime/hip/lib/libamdhip64.so` and `runtime/rocr/lib/
libhsa-runtime64.so` exist (unversioned, no suffix). It prepends both to
`LD_LIBRARY_PATH` and `ldd`-checks that the built `libllama-common.so` carries
the "Qwen4Exp MTP image position" marker and the HIP lib carries the
`flash_attn_index_mask_clear/set/empty` symbols before exec.
The build links the venv through RUNPATH, so those two files are an existence
gate, not a real dependency. `serve-frank-large.sh` satisfies the gate
idempotently: it finds the venv's unversioned devel symlinks
(`.venv-rocm/lib/python*/site-packages/_rocm_sdk_devel/lib/`) and `ln -sfn`-s
both into `runtime/{hip,rocr}/lib/`. **The first deploy of this endpoint failed
exactly here** — the build linked the venv but `runtime/` did not exist, so the
launcher exited 2 and the unit crash-looped until the serve script created the
symlinks.
## Payload
`scripts/download-frank-large-orca.sh` assembles `jcbtc/Qwen3.8-Flash-CIRU-
STRIX-Orca` (HuggingFace, public and ungated) into
`/home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca/`, sha256-verified. The
launcher refuses to start without the main GGUF, the MTP head and the PLE
sidecar:
| file | bytes | role |
|---|---|---|
| `Qwen3.8-Flash-CIRU-STRIX-Orca.gguf` | 79,397,818,912 | the main model (Orca variant) |
| `mtp/Qwen3.8-Flash-CIRU-STRIX-Orca-MTP-Q8_0.gguf` | 4,135,893,440 | the MTP drafter head |
| `ple/ple.payload.bin` | 52,429,053,952 | the per-layer-embedding (PLE) payload — mmap'd and paged, not resident |
| `ple/ple.manifest.json` | — | PLE manifest |
| `ple/ple.scale.bf16` | — | PLE scale |
| `vision/mmproj-Qwen3.8-Flash-Orca-F16.mmproj` | 904,004,320 | the F16 vision projector (on disk, `ENABLE_VISION=0`) |
The main GGUF ships as **transport parts** that the download script assembles
and verifies; the PLE is the large one (52.4 GB) and is paged by the engine's
own workers, not held in RAM.
## Deployed config
Tunables live in `scripts/frank-large.env`; the launcher owns the flag list
(`-sm none` one iGPU no split, `-ngl all`, f16 KV, flash attention,
cont-batching, the PLE sidecar, the MTP drafter, the production kernel-bundle
profile). The departures worth naming:
| flag | value | why |
|---|---|---|
| `CONTEXT_SIZE` | `262144` | the model's FULL trained window (the operator's "full context" call); the GGUF carries no rope scaling |
| `PARALLEL_SLOTS` | `1` | one stream — and the MTP drafter **requires exactly one slot** in this runtime, so the operator's ask and the drafter's constraint are the same number |
| `ENABLE_MTP` | `1` | the MTP drafter, depth 4 (the Orca variant's own `mtp_depth_default`) |
| `ENABLE_VISION` | `0` | text-only for now; the F16 projector is on disk, one flip to enable. The fleet's image path is [[frank-small-gemma4-12b]] either way |
| `PLE_CACHE_MIB` | `4096` | the PLE pager's hot window kept resident; the vendor production-profile default. The 52.4 GB payload is mmap'd and paged around it |
| `HOST` | `0.0.0.0` | the launcher consumes it for `--host`; the value doubles as a JIT-cache directory name, which is why `0.0.0.0` is safe (HIP reads `HOST` bare — see the AGENTS.md note) |
`TEMP`, `HOST` and `GPU_MAX_HEAP_SIZE` are the three names to avoid for any
new tunable: HIP reads them bare.
## MTP
The Multi-Token Prediction drafter is a **separate Q8_0 head** (not inside the
main GGUF), loaded by the launcher. It requires exactly one slot, which is why
`PARALLEL_SLOTS=1`. Depth 4 is the Orca variant's default.
Verified at deploy (2026-09-19): the per-task `draft acceptance = 0.65625
(21/32), mean len = 3.62` — the drafter is engaged and accepting, not silently
standing down.
## PLE
The per-layer-embedding payload (52.4 GB) is the Qwen3.8-Flash-Next
architecture's signature: token embeddings live outside the main weight set and
are paged on demand by the engine's own workers (`GGML_QWEN4EXP_PLE_WORKERS=16`,
`GGML_QWEN4EXP_PLE_IO_WORKERS=32` in the production profile). `PLE_CACHE_MIB=4096`
is the hot window kept resident; the rest is mmap'd file cache. **Do not treat
the 52.4 GB as resident memory** — `free` and the dashboards will not show it
pinned, and that is correct.
## Client contract
The differences from a stock llama.cpp endpoint that bite:
- **`reasoning_content`** carries the reasoning trace (the standard OpenAI
field). The serve script passes `--chat-template-kwargs
'{"preserve_thinking":true}'` so prior thinking blocks are kept across turns;
a request-level `chat_template_kwargs` still wins.
- **`reasoning_effort` is gated.** Qwen3.8's template accepts only
`xhigh`, `medium`, `low` and **500s on anything else** (not a clamp). The
harness folds every level onto that set in `models.yml`
(`reasoningEffortMap`: minimal/low→low, medium→medium, high/max→xhigh) —
subagents run at `high`, which is why the map exists.
- **Text only.** `ENABLE_VISION=0`, so image content parts are refused. The
fleet's image path is [[frank-small-gemma4-12b]].
- **Sampling.** The server has no sampler defaults of its own; the harness
applies the per-model recipe from `models.yml` (temp 1.0 / top_p 0.95 / top_k
20 / min_p 0.0). A client that stays silent gets the launcher's default.
- **`tools` works** — the launcher passes `--jinja`, and the Qwen3.8 template
supplies the tool-call grammar. Verified at deploy: a tool call returns
`tool_calls`.
## Deploy and operate
```bash
./scripts/deploy.sh dankclaw --service frank-large
```
The handler pulls the checkout (ff-only), (re)links the unit, and gates on
`/health`. First boot is the slow one (model load ~46 s + graph capture); a
warm restart is fast.
```bash
sudo systemctl status frank-large # the root unit
sudo journalctl -u frank-large -n 60 # startup account: runtime check, model load, "listening on http://0.0.0.0:7770", MTP draft loaded
curl -s http://127.0.0.1:7770/health # ok
curl -s http://127.0.0.1:7770/v1/models # Qwen3.8-Flash-CIRU-STRIX-Orca
```
**Stop and mask before any GPU experiment** (the AGENTS.md bench protocol):
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`. The
frank-large serve process is a `llama-server` **and so is frank-small on the
same host** — never `pkill -x llama-server`. If you must kill by pattern, use
`pkill -f 'Qwen3.8-Flash-CIRU-STRIX-Orca'` (the model path is unique to this
endpoint and does not appear in the deploying shell's own command line).
## Troubleshooting
| symptom | cause |
|---|---|
| launcher exits 2: "The v4.4 HIP/ROCr runtime is missing" | `runtime/{hip,rocr}/lib/*.so` absent — the serve script's `ln -sfn` did not run, or the venv was moved. Re-run the serve script or rebuild: `bash $RUNTIME_DIR/scripts/ciru/setup-linux-amd.sh --install-host-deps` |
| "missing SDK venv runtime under .../.venv-rocm" | the venv's unversioned devel symlinks are gone — rebuild the runtime |
| "missing target model / missing payload" | a download did not finish — re-run `scripts/download-frank-large-orca.sh` |
| loads, `/health` ok, first completion slow (~45-60 s) | the PLE pager pulling the hot window in from file cache — not a crash, do not restart over it |
| a subagent 500s naming `reasoning_effort` | the Qwen3.8 template gate — the harness `reasoningEffortMap` must fold the level onto {xhigh, medium, low} |
| `free` / dashboards do not show 52 GB pinned | correct — the PLE is mmap'd file cache, not resident; only `PLE_CACHE_MIB` is hot |
Retrieval: this is a **root** unit, so `journalctl -u frank-large` is the
durable record (no `--user`, no `sudo` needed for the journal once the unit is
readable). There is no container log — there is no container.
## Rollback
The predecessor is intact: the halogen payload stays at
`/home/dank/models/halogen` (retired, not deleted), and the halogen quadlet is
archived under `.AGENTTRASH/`. Rolling back is a `git revert` of this change set
plus a deploy — **but the root unit and the halogen quadlet both claim port
7770**, so the handler's port-free gate covers that race. See
[[halogen-flash-server]].
The `amd_iommu=off` kernel argument is separate and reverts on its own — see
[[dankclaw-iommu-translation-off]].
## Citations
- [jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca](https://huggingface.co/jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca) — weights, MTP head, PLE payload, and the v4.4.1 QSA window-hotfix source tarball
- [CIRU runtime v4.4.1](https://github.com/ciru) — `scripts/ciru/run-server.sh` (runtime gate), `profiles/strix-halo-production.env` (kernel bundle), `docs/BUILD_LINUX.md`
@@ -1,7 +1,7 @@
---
type: configuration
title: frank-medium llama.cpp Runbook (MTP drafter)
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the model's own MTP head (the z-lab DFlash2 sidecar kept on disk, one env change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a 35:30 split (skewed toward the x16-slot card; see the 2026-09-17 part 3 section), a 393216-position KV pool (196608 per conversation over two slots), vision via the BF16 mmproj.
title: frank-medium llama.cpp Runbook (DFlash2 drafter)
description: The ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on Dankserver serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a balanced 32:33 split, a 327680-position KV pool (163840 per conversation over two slots), f16 KV plus a 16 GiB host-RAM prompt cache, vision via the BF16 mmproj.
tags:
- frank-medium
- dankserver
@@ -12,16 +12,19 @@ tags:
- gfx1201
- r9700
- layer-split
timestamp: 2026-09-17
timestamp: 2026-09-19
---
# frank-medium — llama.cpp (DFlash2 drafter)
The ACTIVE frank-medium deployment since 2026-09-16: **mainline llama.cpp**
serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at `UD-Q8_K_XL`
with the **z-lab DFlash2** block-diffusion drafter, as **one process across both
Radeon AI PRO R9700 cards**, layer-split. DSpark was tried in the drafter's
place for a few hours the same day and reverted — see *The DSpark excursion*.
The ACTIVE frank-medium deployment since 2026-09-16 (model swapped to
orcarouter Q8_0 on 2026-09-19): **mainline llama.cpp**
serving `orcarouter/Qwen3.8-27B-Uncensored-GGUF` at `Q8_0`
with the **z-lab DFlash2** `Q4_K_M` block-diffusion drafter, as **one process across both
Radeon AI PRO R9700 cards**, layer-split at a balanced 32:33. The predecessor
model (ajgazin Swift-Qwen3.8-27B UD-Q8_K_XL, 35:30 skew, 393216 pool) is the
calibration history in the sizing tables below. DSpark was tried in the drafter's
place on 2026-09-16 and reverted — see *The DSpark excursion*.
It replaced the vllm-radiance MXFP4 replica pair + routing proxy
([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in
the same day the pair went live. It is also the box's **only** LLM runtime: the
+32 -26
View File
@@ -1,13 +1,13 @@
# LLM Fleet — Knowledge Base
Documentation for the LLM inference fleet on the two hosts (2026-09-16):
Documentation for the LLM inference fleet on the two hosts (2026-09-19):
**mainline llama.cpp on Dankserver** serving frank-medium as one layer-split
process across both Radeon AI PRO R9700 cards (ajgazin Swift-Qwen3.8-27B-
Uncensored-Dynamic-MTP UD-Q8_K_XL + the z-lab DFlash2 drafter), and TWO
endpoints on Dankclaw — halogen-flash-server (frank-large, Qwen3.8-Flash-Next
w4b on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small,
ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with MTP and vision on the RTX 3090, port
7771).
process across both Radeon AI PRO R9700 cards (orcarouter Qwen3.8-27B-
Uncensored Q8_0 + the z-lab DFlash2 Q4_K_M drafter, balanced 32:33), and TWO
endpoints on Dankclaw — the CIRU v4.4.1 runtime (frank-large, jcbtc
Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU, port 7770) and mainline
llama.cpp (frank-small, HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on
the RTX 3090, port 7771).
> 2026-08-22: the former `vllmScripts` repository is merged into the
> serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -23,22 +23,22 @@ graph LR
gfx1201, 32 GiB each]
B2[llama.cpp, layer split
frank-medium, port 7770]
C[Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL
ONE process on both cards, 262144 ctx / 1 slot, DFlash2]
C[orcarouter Qwen3.8-27B Q8_0 + DFlash2 Q4_K_M
ONE process on both cards, 2x163840 ctx, balanced 32:33]
A --> B2 --> C
end
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
E2[RTX 3090 (PCIe x4 dock) +
gfx1151 iGPU 64 GiB unified memory]
E[halogen-flash-server 0.5.8
E[CIRU v4.4.1 runtime
frank-large, port 7770]
F[peonist-ai Qwen3.8-Flash-Next w4b
2 x 200192 ctx over one 400384-position pool]
F[jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b
262144 ctx / 1 slot, MTP depth 4 + PLE]
G[mainline llama.cpp 37b53fd
frank-small, port 7771]
H[Qwen3.8-27B GSQ-RCO IQ3_S
131072 ctx / 1 slot, MTP + vision]
H[Gemma4-12B QAT Q4_K_M
4 x 262144 ctx, MTP + vision + audio]
E2 --> E --> F
E2 --> G --> H
end
@@ -57,11 +57,12 @@ frank-small, port 7771]
## Active Deployment
- [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since
2026-09-16: mainline llama.cpp serving
ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF at UD-Q8_K_XL with the
z-lab DFlash2 Q8_0 drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE
layer-split process behind :7770 (tensor parallelism is impossible on this
host — the second card cannot dispatch an RCCL collective)
2026-09-16 (model swapped 2026-09-19): mainline llama.cpp serving
orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M
drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE layer-split process
behind :7770 (tensor parallelism is impossible on this host — the second card
cannot dispatch an RCCL collective), balanced 32:33 over two 163,840-token
slots
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16, one day after it was
deployed: the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 +
ggz14/radiance-vllm-mxfp4 patches) serving
@@ -81,12 +82,17 @@ frank-small, port 7771]
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
- [[deployment]] — deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[halogen-flash-server]] — ACTIVE frank-large runbook again since 2026-09-16
(rootless Quadlet, port 7770): peonist-ai halogen-flash-server 0.5.8 serving
peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn + quality
overlay) on the gfx1151 iGPU — two 200k conversations over one
- [[frank-large-ciru-orca]] — ACTIVE frank-large runbook since 2026-09-19
(root systemd unit, port 7770): the CIRU v4.4.1 runtime (private host build,
QSA hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151
iGPU — one 262,144-token conversation, MTP depth 4, 4096 MiB PLE cache,
runtime-root gate, client contract, troubleshooting
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
[[frank-large-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
400,384-position pool, deployed flags, memory model, client contract,
troubleshooting
troubleshooting (2026-09-16 → 09-19)
- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
(root systemd unit, port 7771): mainline llama.cpp serving
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
@@ -140,8 +146,8 @@ frank-small, port 7771]
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
(deployed and reverted 2026-08-22)
- [[Dankserver System Specs]] — the 2x R9700 box (frank-medium)
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on
halogen-flash-server since 2026-09-16, frank-small on the RTX 3090)
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on the
CIRU v4.4.1 runtime since 2026-09-19, frank-small on the RTX 3090)
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
the retirement record — the IOMMU/SVA err=-19 root cause and where
+17 -11
View File
@@ -116,12 +116,12 @@ The LLM inference fleet on the two hosts. Full index:
- [[deployment]] — unified deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since
2026-09-16: mainline llama.cpp (gfx1201 HIP build) serving
ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF at UD-Q8_K_XL with
the DFlash2 drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF at Q8_0) on
Dankserver's 2x Radeon AI PRO R9700 as one layer-split process behind the
:7770 endpoint — no tensor parallelism, because the second card cannot
dispatch an RCCL collective
2026-09-16 (model swapped 2026-09-19): mainline llama.cpp (gfx1201 HIP build)
serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the DFlash2
drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF at Q4_K_M) on Dankserver's 2x Radeon
AI PRO R9700 as one layer-split process behind the :7770 endpoint, balanced
32:33 over two 163,840-token slots — no tensor parallelism, because the
second card cannot dispatch an RCCL collective
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16: the vllm-radiance
MXFP4 stack serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on
Dankserver's 2x Radeon AI PRO R9700 as two single-GPU replicas behind a
@@ -145,11 +145,17 @@ The LLM inference fleet on the two hosts. Full index:
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
rollback record)
- [[halogen-flash-server]] — the ACTIVE frank-large runbook again since
2026-09-16 (rootless Quadlet, port 7770): peonist-ai/halogen-flash-server
0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (w4b `.hgn` + quality
overlay) on the gfx1151 iGPU — two 200k conversations over one
400,384-position pool, deployed flags, memory model, client contract
- [[frank-large-ciru-orca]] — the ACTIVE frank-large runbook since 2026-09-19
(root systemd unit, port 7770): the CIRU v4.4.1 runtime (private host build,
QSA hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151
iGPU — one 262,144-token conversation, MTP depth 4, 4096 MiB PLE cache,
runtime-root gate, client contract
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
[[frank-large-ciru-orca]]; rollback target, 2026-09-16 → 09-19):
peonist-ai/halogen-flash-server 0.5.8 serving peonist-ai/halogen-qwen3.8-
flash-next (w4b `.hgn` + quality overlay) on the gfx1151 iGPU — two 200k
conversations over one 400,384-position pool, deployed flags, memory model,
client contract
- [[frank-large-ik-uncensored]] — the RETIRED 2026-09-16 frank-large runbook
and rollback target (2026-09-15 → 09-16): ik_llama.cpp serving
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 +