docs: wrap up the 2026-09-19 fleet cutover
frank-large -> CIRU v4.4.1 Orca (new runbook), frank-medium -> the deployed
orcarouter Q8_0 / DFlash2 Q4_K_M / balanced 32:33 / 327680 state. The d374ab9
frank-medium deploy updated only scripts/tests and left AGENTS.md, the runbook
and both indexes on the predecessor (ajgazin Swift UD-Q8_K_XL, 35:30, 393216).
- AGENTS.md: frank-large section (CIRU Orca, root unit, runtime gate, PLE, MTP)
+ frank-medium section and topology row to the deployed shape
- new runbook frank-large-ciru-orca.md
- frank-medium runbook header/intro: orcarouter Q8_0 swap, 32:33, 327680;
predecessor kept as calibration history
- LLM index.md: intro + mermaid (halogen->CIRU, GSQ-RCO->Gemma4, Swift->orcarouter),
add ciru-orca, retire halogen
- top-level index.md: add ciru-orca link (docs-index seam), retire halogen,
fix frank-medium entry
This commit is contained in:
@@ -0,0 +1,219 @@
|
||||
---
|
||||
type: reference
|
||||
title: CIRU v4.4.1 Orca on Dankclaw (frank-large)
|
||||
description: The active frank-large runbook — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing.
|
||||
tags:
|
||||
- llm
|
||||
- ciru
|
||||
- dankclaw
|
||||
- frank-large
|
||||
- strix-halo
|
||||
- gfx1151
|
||||
- qwen4exp
|
||||
- rocm
|
||||
- mtp
|
||||
timestamp: 2026-09-19
|
||||
---
|
||||
|
||||
# CIRU v4.4.1 Orca on Dankclaw (frank-large)
|
||||
|
||||
> **ACTIVE since 2026-09-19.** It replaced halogen-flash-server (see
|
||||
> [[halogen-flash-server]], the current rollback), which had replaced
|
||||
> ik_llama.cpp (see [[frank-large-ik-uncensored]]). Deployed as a **root
|
||||
> systemd unit** (`scripts/frank-large.service`, `User=dank`), port 7770, on the
|
||||
> same gfx1151 iGPU the halogen quadlet used. The RTX 3090 it does not touch
|
||||
> belongs to [[frank-small-gemma4-12b]].
|
||||
|
||||
CIRU is a llama.cpp fork tuned for the Qwen3.8-Flash-Next ("qwen4exp")
|
||||
architecture on AMD gfx1151 (Strix Halo). v4.4.1 carries a **QSA window
|
||||
hotfix** that is what makes vision and the MTP drafter work on this build; the
|
||||
pre-applied hotfix source ships in the model repo under
|
||||
`runtime/v4.4.1/qsa-window-hotfix/`. Like every other fleet endpoint it is a
|
||||
llama.cpp runtime — `llama-bench` is its own harness.
|
||||
|
||||
## Shape
|
||||
|
||||
One root unit, one process, one published port:
|
||||
|
||||
| half | port | reachable |
|
||||
|---|---|---|
|
||||
| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
|
||||
|
||||
Unlike the halogen predecessor (two ports: an unauthenticated engine on 8730
|
||||
and an OpenAI front-end on 8731), CIRU is a single `llama-server` process —
|
||||
there is no separate engine port to keep private.
|
||||
|
||||
`/v1/models` reports the loaded model name, **`Qwen3.8-Flash-CIRU-STRIX-Orca`**
|
||||
(the launcher sets no `--alias`). The endpoint serves **whatever model name a
|
||||
request names** — llama.cpp does not reject an unknown id — so the harness's
|
||||
`frank-large` label in `models.yml` works without a rename.
|
||||
|
||||
## Runtime
|
||||
|
||||
A **private host build**, not a container image. The v4.4.1 source with the QSA
|
||||
window hotfix, built on the host against the private ROCm SDK that
|
||||
`scripts/ciru/setup-linux-amd.sh` installs into the checkout's own venv:
|
||||
|
||||
| path | what |
|
||||
|---|---|
|
||||
| `/home/dank/GitPackages/ciru-runtime-v4.4.1-qsa-hotfix/` | the checkout (source + build + venv) |
|
||||
| `build-gfx1151-sdk/bin/llama-server` | the built binary |
|
||||
| `.venv-rocm/` | the private ROCm SDK venv (~5.7 GB); the binary resolves its libs through RUNPATH |
|
||||
| `scripts/ciru/run-orca-server.sh` | the launcher (`exec`'d by the serve script) |
|
||||
| `profiles/strix-halo-production.env` | the 36-variable kernel bundle + PLE pager defaults, sourced by the launcher AFTER `frank-large.env` (every line is `${VAR:-default}`, so a value set in `frank-large.env` wins) |
|
||||
|
||||
### The runtime gate
|
||||
|
||||
`run-orca-server.sh` sources `run-server.sh`, which **exits 2** unless
|
||||
`${repo_root}/runtime/hip/lib/libamdhip64.so` and `runtime/rocr/lib/
|
||||
libhsa-runtime64.so` exist (unversioned, no suffix). It prepends both to
|
||||
`LD_LIBRARY_PATH` and `ldd`-checks that the built `libllama-common.so` carries
|
||||
the "Qwen4Exp MTP image position" marker and the HIP lib carries the
|
||||
`flash_attn_index_mask_clear/set/empty` symbols before exec.
|
||||
|
||||
The build links the venv through RUNPATH, so those two files are an existence
|
||||
gate, not a real dependency. `serve-frank-large.sh` satisfies the gate
|
||||
idempotently: it finds the venv's unversioned devel symlinks
|
||||
(`.venv-rocm/lib/python*/site-packages/_rocm_sdk_devel/lib/`) and `ln -sfn`-s
|
||||
both into `runtime/{hip,rocr}/lib/`. **The first deploy of this endpoint failed
|
||||
exactly here** — the build linked the venv but `runtime/` did not exist, so the
|
||||
launcher exited 2 and the unit crash-looped until the serve script created the
|
||||
symlinks.
|
||||
|
||||
## Payload
|
||||
|
||||
`scripts/download-frank-large-orca.sh` assembles `jcbtc/Qwen3.8-Flash-CIRU-
|
||||
STRIX-Orca` (HuggingFace, public and ungated) into
|
||||
`/home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca/`, sha256-verified. The
|
||||
launcher refuses to start without the main GGUF, the MTP head and the PLE
|
||||
sidecar:
|
||||
|
||||
| file | bytes | role |
|
||||
|---|---|---|
|
||||
| `Qwen3.8-Flash-CIRU-STRIX-Orca.gguf` | 79,397,818,912 | the main model (Orca variant) |
|
||||
| `mtp/Qwen3.8-Flash-CIRU-STRIX-Orca-MTP-Q8_0.gguf` | 4,135,893,440 | the MTP drafter head |
|
||||
| `ple/ple.payload.bin` | 52,429,053,952 | the per-layer-embedding (PLE) payload — mmap'd and paged, not resident |
|
||||
| `ple/ple.manifest.json` | — | PLE manifest |
|
||||
| `ple/ple.scale.bf16` | — | PLE scale |
|
||||
| `vision/mmproj-Qwen3.8-Flash-Orca-F16.mmproj` | 904,004,320 | the F16 vision projector (on disk, `ENABLE_VISION=0`) |
|
||||
|
||||
The main GGUF ships as **transport parts** that the download script assembles
|
||||
and verifies; the PLE is the large one (52.4 GB) and is paged by the engine's
|
||||
own workers, not held in RAM.
|
||||
|
||||
## Deployed config
|
||||
|
||||
Tunables live in `scripts/frank-large.env`; the launcher owns the flag list
|
||||
(`-sm none` one iGPU no split, `-ngl all`, f16 KV, flash attention,
|
||||
cont-batching, the PLE sidecar, the MTP drafter, the production kernel-bundle
|
||||
profile). The departures worth naming:
|
||||
|
||||
| flag | value | why |
|
||||
|---|---|---|
|
||||
| `CONTEXT_SIZE` | `262144` | the model's FULL trained window (the operator's "full context" call); the GGUF carries no rope scaling |
|
||||
| `PARALLEL_SLOTS` | `1` | one stream — and the MTP drafter **requires exactly one slot** in this runtime, so the operator's ask and the drafter's constraint are the same number |
|
||||
| `ENABLE_MTP` | `1` | the MTP drafter, depth 4 (the Orca variant's own `mtp_depth_default`) |
|
||||
| `ENABLE_VISION` | `0` | text-only for now; the F16 projector is on disk, one flip to enable. The fleet's image path is [[frank-small-gemma4-12b]] either way |
|
||||
| `PLE_CACHE_MIB` | `4096` | the PLE pager's hot window kept resident; the vendor production-profile default. The 52.4 GB payload is mmap'd and paged around it |
|
||||
| `HOST` | `0.0.0.0` | the launcher consumes it for `--host`; the value doubles as a JIT-cache directory name, which is why `0.0.0.0` is safe (HIP reads `HOST` bare — see the AGENTS.md note) |
|
||||
|
||||
`TEMP`, `HOST` and `GPU_MAX_HEAP_SIZE` are the three names to avoid for any
|
||||
new tunable: HIP reads them bare.
|
||||
|
||||
## MTP
|
||||
|
||||
The Multi-Token Prediction drafter is a **separate Q8_0 head** (not inside the
|
||||
main GGUF), loaded by the launcher. It requires exactly one slot, which is why
|
||||
`PARALLEL_SLOTS=1`. Depth 4 is the Orca variant's default.
|
||||
|
||||
Verified at deploy (2026-09-19): the per-task `draft acceptance = 0.65625
|
||||
(21/32), mean len = 3.62` — the drafter is engaged and accepting, not silently
|
||||
standing down.
|
||||
|
||||
## PLE
|
||||
|
||||
The per-layer-embedding payload (52.4 GB) is the Qwen3.8-Flash-Next
|
||||
architecture's signature: token embeddings live outside the main weight set and
|
||||
are paged on demand by the engine's own workers (`GGML_QWEN4EXP_PLE_WORKERS=16`,
|
||||
`GGML_QWEN4EXP_PLE_IO_WORKERS=32` in the production profile). `PLE_CACHE_MIB=4096`
|
||||
is the hot window kept resident; the rest is mmap'd file cache. **Do not treat
|
||||
the 52.4 GB as resident memory** — `free` and the dashboards will not show it
|
||||
pinned, and that is correct.
|
||||
|
||||
## Client contract
|
||||
|
||||
The differences from a stock llama.cpp endpoint that bite:
|
||||
|
||||
- **`reasoning_content`** carries the reasoning trace (the standard OpenAI
|
||||
field). The serve script passes `--chat-template-kwargs
|
||||
'{"preserve_thinking":true}'` so prior thinking blocks are kept across turns;
|
||||
a request-level `chat_template_kwargs` still wins.
|
||||
- **`reasoning_effort` is gated.** Qwen3.8's template accepts only
|
||||
`xhigh`, `medium`, `low` and **500s on anything else** (not a clamp). The
|
||||
harness folds every level onto that set in `models.yml`
|
||||
(`reasoningEffortMap`: minimal/low→low, medium→medium, high/max→xhigh) —
|
||||
subagents run at `high`, which is why the map exists.
|
||||
- **Text only.** `ENABLE_VISION=0`, so image content parts are refused. The
|
||||
fleet's image path is [[frank-small-gemma4-12b]].
|
||||
- **Sampling.** The server has no sampler defaults of its own; the harness
|
||||
applies the per-model recipe from `models.yml` (temp 1.0 / top_p 0.95 / top_k
|
||||
20 / min_p 0.0). A client that stays silent gets the launcher's default.
|
||||
- **`tools` works** — the launcher passes `--jinja`, and the Qwen3.8 template
|
||||
supplies the tool-call grammar. Verified at deploy: a tool call returns
|
||||
`tool_calls`.
|
||||
|
||||
## Deploy and operate
|
||||
|
||||
```bash
|
||||
./scripts/deploy.sh dankclaw --service frank-large
|
||||
```
|
||||
|
||||
The handler pulls the checkout (ff-only), (re)links the unit, and gates on
|
||||
`/health`. First boot is the slow one (model load ~46 s + graph capture); a
|
||||
warm restart is fast.
|
||||
|
||||
```bash
|
||||
sudo systemctl status frank-large # the root unit
|
||||
sudo journalctl -u frank-large -n 60 # startup account: runtime check, model load, "listening on http://0.0.0.0:7770", MTP draft loaded
|
||||
curl -s http://127.0.0.1:7770/health # ok
|
||||
curl -s http://127.0.0.1:7770/v1/models # Qwen3.8-Flash-CIRU-STRIX-Orca
|
||||
```
|
||||
|
||||
**Stop and mask before any GPU experiment** (the AGENTS.md bench protocol):
|
||||
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`. The
|
||||
frank-large serve process is a `llama-server` **and so is frank-small on the
|
||||
same host** — never `pkill -x llama-server`. If you must kill by pattern, use
|
||||
`pkill -f 'Qwen3.8-Flash-CIRU-STRIX-Orca'` (the model path is unique to this
|
||||
endpoint and does not appear in the deploying shell's own command line).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| symptom | cause |
|
||||
|---|---|
|
||||
| launcher exits 2: "The v4.4 HIP/ROCr runtime is missing" | `runtime/{hip,rocr}/lib/*.so` absent — the serve script's `ln -sfn` did not run, or the venv was moved. Re-run the serve script or rebuild: `bash $RUNTIME_DIR/scripts/ciru/setup-linux-amd.sh --install-host-deps` |
|
||||
| "missing SDK venv runtime under .../.venv-rocm" | the venv's unversioned devel symlinks are gone — rebuild the runtime |
|
||||
| "missing target model / missing payload" | a download did not finish — re-run `scripts/download-frank-large-orca.sh` |
|
||||
| loads, `/health` ok, first completion slow (~45-60 s) | the PLE pager pulling the hot window in from file cache — not a crash, do not restart over it |
|
||||
| a subagent 500s naming `reasoning_effort` | the Qwen3.8 template gate — the harness `reasoningEffortMap` must fold the level onto {xhigh, medium, low} |
|
||||
| `free` / dashboards do not show 52 GB pinned | correct — the PLE is mmap'd file cache, not resident; only `PLE_CACHE_MIB` is hot |
|
||||
|
||||
Retrieval: this is a **root** unit, so `journalctl -u frank-large` is the
|
||||
durable record (no `--user`, no `sudo` needed for the journal once the unit is
|
||||
readable). There is no container log — there is no container.
|
||||
|
||||
## Rollback
|
||||
|
||||
The predecessor is intact: the halogen payload stays at
|
||||
`/home/dank/models/halogen` (retired, not deleted), and the halogen quadlet is
|
||||
archived under `.AGENTTRASH/`. Rolling back is a `git revert` of this change set
|
||||
plus a deploy — **but the root unit and the halogen quadlet both claim port
|
||||
7770**, so the handler's port-free gate covers that race. See
|
||||
[[halogen-flash-server]].
|
||||
|
||||
The `amd_iommu=off` kernel argument is separate and reverts on its own — see
|
||||
[[dankclaw-iommu-translation-off]].
|
||||
|
||||
## Citations
|
||||
|
||||
- [jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca](https://huggingface.co/jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca) — weights, MTP head, PLE payload, and the v4.4.1 QSA window-hotfix source tarball
|
||||
- [CIRU runtime v4.4.1](https://github.com/ciru) — `scripts/ciru/run-server.sh` (runtime gate), `profiles/strix-halo-production.env` (kernel bundle), `docs/BUILD_LINUX.md`
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: configuration
|
||||
title: frank-medium llama.cpp Runbook (MTP drafter)
|
||||
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the model's own MTP head (the z-lab DFlash2 sidecar kept on disk, one env change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a 35:30 split (skewed toward the x16-slot card; see the 2026-09-17 part 3 section), a 393216-position KV pool (196608 per conversation over two slots), vision via the BF16 mmproj.
|
||||
title: frank-medium llama.cpp Runbook (DFlash2 drafter)
|
||||
description: The ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on Dankserver serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a balanced 32:33 split, a 327680-position KV pool (163840 per conversation over two slots), f16 KV plus a 16 GiB host-RAM prompt cache, vision via the BF16 mmproj.
|
||||
tags:
|
||||
- frank-medium
|
||||
- dankserver
|
||||
@@ -12,16 +12,19 @@ tags:
|
||||
- gfx1201
|
||||
- r9700
|
||||
- layer-split
|
||||
timestamp: 2026-09-17
|
||||
timestamp: 2026-09-19
|
||||
---
|
||||
|
||||
# frank-medium — llama.cpp (DFlash2 drafter)
|
||||
|
||||
The ACTIVE frank-medium deployment since 2026-09-16: **mainline llama.cpp**
|
||||
serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at `UD-Q8_K_XL`
|
||||
with the **z-lab DFlash2** block-diffusion drafter, as **one process across both
|
||||
Radeon AI PRO R9700 cards**, layer-split. DSpark was tried in the drafter's
|
||||
place for a few hours the same day and reverted — see *The DSpark excursion*.
|
||||
The ACTIVE frank-medium deployment since 2026-09-16 (model swapped to
|
||||
orcarouter Q8_0 on 2026-09-19): **mainline llama.cpp**
|
||||
serving `orcarouter/Qwen3.8-27B-Uncensored-GGUF` at `Q8_0`
|
||||
with the **z-lab DFlash2** `Q4_K_M` block-diffusion drafter, as **one process across both
|
||||
Radeon AI PRO R9700 cards**, layer-split at a balanced 32:33. The predecessor
|
||||
model (ajgazin Swift-Qwen3.8-27B UD-Q8_K_XL, 35:30 skew, 393216 pool) is the
|
||||
calibration history in the sizing tables below. DSpark was tried in the drafter's
|
||||
place on 2026-09-16 and reverted — see *The DSpark excursion*.
|
||||
It replaced the vllm-radiance MXFP4 replica pair + routing proxy
|
||||
([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in
|
||||
the same day the pair went live. It is also the box's **only** LLM runtime: the
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-09-16):
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-09-19):
|
||||
**mainline llama.cpp on Dankserver** serving frank-medium as one layer-split
|
||||
process across both Radeon AI PRO R9700 cards (ajgazin Swift-Qwen3.8-27B-
|
||||
Uncensored-Dynamic-MTP UD-Q8_K_XL + the z-lab DFlash2 drafter), and TWO
|
||||
endpoints on Dankclaw — halogen-flash-server (frank-large, Qwen3.8-Flash-Next
|
||||
w4b on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small,
|
||||
ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with MTP and vision on the RTX 3090, port
|
||||
7771).
|
||||
process across both Radeon AI PRO R9700 cards (orcarouter Qwen3.8-27B-
|
||||
Uncensored Q8_0 + the z-lab DFlash2 Q4_K_M drafter, balanced 32:33), and TWO
|
||||
endpoints on Dankclaw — the CIRU v4.4.1 runtime (frank-large, jcbtc
|
||||
Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU, port 7770) and mainline
|
||||
llama.cpp (frank-small, HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on
|
||||
the RTX 3090, port 7771).
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -23,22 +23,22 @@ graph LR
|
||||
gfx1201, 32 GiB each]
|
||||
B2[llama.cpp, layer split
|
||||
frank-medium, port 7770]
|
||||
C[Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL
|
||||
ONE process on both cards, 262144 ctx / 1 slot, DFlash2]
|
||||
C[orcarouter Qwen3.8-27B Q8_0 + DFlash2 Q4_K_M
|
||||
ONE process on both cards, 2x163840 ctx, balanced 32:33]
|
||||
A --> B2 --> C
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[RTX 3090 (PCIe x4 dock) +
|
||||
gfx1151 iGPU 64 GiB unified memory]
|
||||
E[halogen-flash-server 0.5.8
|
||||
E[CIRU v4.4.1 runtime
|
||||
frank-large, port 7770]
|
||||
F[peonist-ai Qwen3.8-Flash-Next w4b
|
||||
2 x 200192 ctx over one 400384-position pool]
|
||||
F[jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b
|
||||
262144 ctx / 1 slot, MTP depth 4 + PLE]
|
||||
G[mainline llama.cpp 37b53fd
|
||||
frank-small, port 7771]
|
||||
H[Qwen3.8-27B GSQ-RCO IQ3_S
|
||||
131072 ctx / 1 slot, MTP + vision]
|
||||
H[Gemma4-12B QAT Q4_K_M
|
||||
4 x 262144 ctx, MTP + vision + audio]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -57,11 +57,12 @@ frank-small, port 7771]
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since
|
||||
2026-09-16: mainline llama.cpp serving
|
||||
ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF at UD-Q8_K_XL with the
|
||||
z-lab DFlash2 Q8_0 drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE
|
||||
layer-split process behind :7770 (tensor parallelism is impossible on this
|
||||
host — the second card cannot dispatch an RCCL collective)
|
||||
2026-09-16 (model swapped 2026-09-19): mainline llama.cpp serving
|
||||
orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M
|
||||
drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE layer-split process
|
||||
behind :7770 (tensor parallelism is impossible on this host — the second card
|
||||
cannot dispatch an RCCL collective), balanced 32:33 over two 163,840-token
|
||||
slots
|
||||
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16, one day after it was
|
||||
deployed: the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 +
|
||||
ggz14/radiance-vllm-mxfp4 patches) serving
|
||||
@@ -81,12 +82,17 @@ frank-small, port 7771]
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[halogen-flash-server]] — ACTIVE frank-large runbook again since 2026-09-16
|
||||
(rootless Quadlet, port 7770): peonist-ai halogen-flash-server 0.5.8 serving
|
||||
peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn + quality
|
||||
overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
- [[frank-large-ciru-orca]] — ACTIVE frank-large runbook since 2026-09-19
|
||||
(root systemd unit, port 7770): the CIRU v4.4.1 runtime (private host build,
|
||||
QSA hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151
|
||||
iGPU — one 262,144-token conversation, MTP depth 4, 4096 MiB PLE cache,
|
||||
runtime-root gate, client contract, troubleshooting
|
||||
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
|
||||
[[frank-large-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
|
||||
0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b
|
||||
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract,
|
||||
troubleshooting
|
||||
troubleshooting (2026-09-16 → 09-19)
|
||||
- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
|
||||
(root systemd unit, port 7771): mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||||
@@ -140,8 +146,8 @@ frank-small, port 7771]
|
||||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the 2x R9700 box (frank-medium)
|
||||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on
|
||||
halogen-flash-server since 2026-09-16, frank-small on the RTX 3090)
|
||||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on the
|
||||
CIRU v4.4.1 runtime since 2026-09-19, frank-small on the RTX 3090)
|
||||
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
|
||||
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
|
||||
the retirement record — the IOMMU/SVA err=-19 root cause and where
|
||||
|
||||
@@ -116,12 +116,12 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[deployment]] — unified deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since
|
||||
2026-09-16: mainline llama.cpp (gfx1201 HIP build) serving
|
||||
ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF at UD-Q8_K_XL with
|
||||
the DFlash2 drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF at Q8_0) on
|
||||
Dankserver's 2x Radeon AI PRO R9700 as one layer-split process behind the
|
||||
:7770 endpoint — no tensor parallelism, because the second card cannot
|
||||
dispatch an RCCL collective
|
||||
2026-09-16 (model swapped 2026-09-19): mainline llama.cpp (gfx1201 HIP build)
|
||||
serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the DFlash2
|
||||
drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF at Q4_K_M) on Dankserver's 2x Radeon
|
||||
AI PRO R9700 as one layer-split process behind the :7770 endpoint, balanced
|
||||
32:33 over two 163,840-token slots — no tensor parallelism, because the
|
||||
second card cannot dispatch an RCCL collective
|
||||
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16: the vllm-radiance
|
||||
MXFP4 stack serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on
|
||||
Dankserver's 2x Radeon AI PRO R9700 as two single-GPU replicas behind a
|
||||
@@ -145,11 +145,17 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
|
||||
Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
|
||||
rollback record)
|
||||
- [[halogen-flash-server]] — the ACTIVE frank-large runbook again since
|
||||
2026-09-16 (rootless Quadlet, port 7770): peonist-ai/halogen-flash-server
|
||||
0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (w4b `.hgn` + quality
|
||||
overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract
|
||||
- [[frank-large-ciru-orca]] — the ACTIVE frank-large runbook since 2026-09-19
|
||||
(root systemd unit, port 7770): the CIRU v4.4.1 runtime (private host build,
|
||||
QSA hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151
|
||||
iGPU — one 262,144-token conversation, MTP depth 4, 4096 MiB PLE cache,
|
||||
runtime-root gate, client contract
|
||||
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
|
||||
[[frank-large-ciru-orca]]; rollback target, 2026-09-16 → 09-19):
|
||||
peonist-ai/halogen-flash-server 0.5.8 serving peonist-ai/halogen-qwen3.8-
|
||||
flash-next (w4b `.hgn` + quality overlay) on the gfx1151 iGPU — two 200k
|
||||
conversations over one 400,384-position pool, deployed flags, memory model,
|
||||
client contract
|
||||
- [[frank-large-ik-uncensored]] — the RETIRED 2026-09-16 frank-large runbook
|
||||
and rollback target (2026-09-15 → 09-16): ik_llama.cpp serving
|
||||
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 +
|
||||
|
||||
Reference in New Issue
Block a user