docs: wrap up the 2026-09-19 fleet cutover
frank-large -> CIRU v4.4.1 Orca (new runbook), frank-medium -> the deployed
orcarouter Q8_0 / DFlash2 Q4_K_M / balanced 32:33 / 327680 state. The d374ab9
frank-medium deploy updated only scripts/tests and left AGENTS.md, the runbook
and both indexes on the predecessor (ajgazin Swift UD-Q8_K_XL, 35:30, 393216).
- AGENTS.md: frank-large section (CIRU Orca, root unit, runtime gate, PLE, MTP)
+ frank-medium section and topology row to the deployed shape
- new runbook frank-large-ciru-orca.md
- frank-medium runbook header/intro: orcarouter Q8_0 swap, 32:33, 327680;
predecessor kept as calibration history
- LLM index.md: intro + mermaid (halogen->CIRU, GSQ-RCO->Gemma4, Swift->orcarouter),
add ciru-orca, retire halogen
- top-level index.md: add ciru-orca link (docs-index seam), retire halogen,
fix frank-medium entry
This commit is contained in:
1 parent
f68bda56c7
commit
764cf41222
5 files changed
+340
-106
No files matched your search
@@ -12,7 +12,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service.
|
||||
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
|
||||
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen-flash-server on
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (CIRU v4.4.1 llama.cpp on
|
||||
the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
|
||||
Dankclaw reverse-proxies both hosts.
|
||||
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
|
||||
@@ -24,8 +24,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (35:30 — skewed toward the x16-slot card; the other sits behind a Gen4 x1 chipset path), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj)|7770|
|
||||
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
|
||||
|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|orcarouter Qwen3.8-27B-Uncensored Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), 2x Radeon AI PRO R9700 in ONE layer-split process (32:33 balanced; ROCm0 Gen4 x16, ROCm1 Gen4 x1 chipset path), 327680 ctx over 2 slots of 163840, batch 6144x512, f16 KV + 16 GiB host-RAM prompt cache, vision (BF16 mmproj)|7770|
|
||||
| Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
@@ -68,8 +68,8 @@ All edits flow through git. Never edit files on a remote machine directly.
|
||||
The planner maps changed paths to services; the remote runner restarts only
|
||||
the selected service and records its revision outside the checkout.
|
||||
Documentation-only changes are a no-op. LLM services are selected only by
|
||||
their own files — frank-large by `dankclaw/frank-large/quadlet/`, frank-small
|
||||
and frank-medium by their unit/env/serve trio in `scripts/` — so unrelated
|
||||
their own files — all three by their unit/env/serve trio in `scripts/` — so
|
||||
unrelated
|
||||
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
|
||||
runs on a host without passwordless sudo; the value never enters the repo.
|
||||
|
||||
@@ -207,9 +207,8 @@ archive.
|
||||
|
||||
**STOP the production service BEFORE benchmarking** — these servers serve
|
||||
real users, and a bench beside a live service starves the GPU and corrupts
|
||||
results. Use the runtime's own harness: `llama-bench` (llama.cpp), `vllm
|
||||
bench` (vLLM), and halogen-flash-server's own `bench`/`sweep` modes for
|
||||
frank-large, which is the active runtime there again. Rolling your own bench
|
||||
results. Use the runtime's own harness: `llama-bench` for every llama.cpp
|
||||
endpoint (all three are llama.cpp now). Rolling your own bench
|
||||
requires asking first.
|
||||
Scratch ports are not used on LLM hosts — a
|
||||
test process cannot share the single GPU anyway; validate against the service
|
||||
@@ -227,11 +226,8 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
|
||||
merely wedged kernel usually still answers ARP.
|
||||
|
||||
1. Stop and mask (self-healing would restart it):
|
||||
- Dankclaw `frank-large` (rootless Quadlet again since 2026-09-16):
|
||||
`systemctl --user stop frank-large`, then move its
|
||||
`~/.config/containers/systemd/frank-large.container` aside and
|
||||
`systemctl --user daemon-reload` — the quadlet generator re-creates that
|
||||
unit from the file on every reload, so the source is the lever, not `mask`.
|
||||
- Dankclaw `frank-large` (root unit, CIRU llama.cpp on the iGPU):
|
||||
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
|
||||
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
|
||||
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
|
||||
- Dankserver `frank-medium` (system unit):
|
||||
@@ -264,42 +260,45 @@ provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
|
||||
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
|
||||
a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-09-16 the image path is `frank-small` (`input: [text, image]`), and
|
||||
`frank-large` is text-only because its halogen deployment loads no vision
|
||||
tower. There is **no audio** input anywhere in the fleet: that capability left
|
||||
with the FLM NPU endpoint on 2026-09-11.
|
||||
`frank-large` is text-only because its CIRU deployment loads no vision
|
||||
projector (the F16 mmproj is on disk, one env flip away). There is **no audio**
|
||||
input anywhere in the fleet: that capability left with the FLM NPU endpoint on
|
||||
2026-09-11.
|
||||
|
||||
### frank-large — halogen-flash-server (Dankclaw, ACTIVE)
|
||||
### frank-large — CIRU v4.4.1 Orca (Dankclaw, ACTIVE)
|
||||
|
||||
halogen-flash-server 0.5.8 as a **rootless Quadlet again since 2026-09-16**
|
||||
(`dankclaw/frank-large/quadlet/frank-large.container`), serving
|
||||
`peonist-ai/halogen-qwen3.8-flash-next` (115.55 GiB w4b `.hgn` checkpoint + a
|
||||
quality overlay that re-fetched to 2.40 GiB on 2026-09-16) on port 7770 from
|
||||
the gfx1151 iGPU. Two conversations of 200,192 tokens resident (pool 400,384,
|
||||
2 slots), greedy default, reasoning extracted to `reasoning_content`, and **no
|
||||
vision tower** — that moved to frank-small. It replaced ik_llama.cpp, which had
|
||||
replaced it for two days.
|
||||
CIRU v4.4.1 (the QSA window hotfix) as a **root systemd unit** (`scripts/
|
||||
frank-large.service`, `User=dank`), serving `jcbtc/Qwen3.8-Flash-CIRU-STRIX-
|
||||
Orca` (79.4 GiB main GGUF + a 4.1 GiB MTP head + a 52.4 GiB PLE payload, all
|
||||
sha256-verified) on port 7770 from the gfx1151 iGPU. One stream, the model's
|
||||
FULL 262,144 trained window (`PARALLEL_SLOTS=1`; the MTP drafter needs exactly
|
||||
one slot in this runtime), MTP depth 4, PLE cache 4096 MiB, reasoning extracted
|
||||
to `reasoning_content`, and **no vision projector** (the F16 mmproj is on disk,
|
||||
`ENABLE_VISION=0` — one env flip to enable). It replaced halogen-flash-server,
|
||||
which had replaced ik_llama.cpp.
|
||||
|
||||
The engine **fetches its own weights on first start** (`HALOGEN_DOWNLOAD`, into
|
||||
the `/home/dank/models/halogen` bind mount): clearing that directory and
|
||||
restarting the unit is the supported way to pick up upstream updates, and it
|
||||
needs the directory to exist **empty** — a missing directory fails the
|
||||
container with `Error: statfs /home/dank/models/halogen: no such file or
|
||||
directory` and the unit loops on `Restart=on-failure` until `mkdir -p` brings
|
||||
it back. The sidecars are versioned in place, so a stale copy is invisible from
|
||||
the file name (both overlays were ~95 MB larger upstream than the copies this
|
||||
host had been serving).
|
||||
The runtime is a **private host build** (`/home/dank/GitPackages/ciru-runtime-
|
||||
v4.4.1-qsa-hotfix/`), not a container image: a gfx1151 HIP build of the CIRU
|
||||
llama.cpp fork with its own bundled ROCm SDK in a venv (`.venv-rocm/`), and the
|
||||
binary resolves the SDK libs via RUNPATH. The launcher (`scripts/ciru/
|
||||
run-orca-server.sh` → `run-server.sh`) has a **runtime gate**: it exits 2 unless
|
||||
`${repo_root}/runtime/hip/lib/libamdhip64.so` and `runtime/rocr/lib/
|
||||
libhsa-runtime64.so` exist. The serve script (`scripts/serve-frank-large.sh`)
|
||||
satisfies that gate idempotently by `ln -sfn`-ing both to the venv's unversioned
|
||||
devel symlinks, then `exec`s the launcher with `--chat-template-kwargs
|
||||
'{"preserve_thinking":true}'` (Qwen3.8's template needs this to carry thinking
|
||||
blocks across turns; a request-level value still wins).
|
||||
|
||||
Measured boot ledger (2026-09-16, `HALOGEN_VERBOSE=1` ships in the unit):
|
||||
device budget `121 − 67.7 − 20 = 33.5 GiB` against a need of `~27.2 GiB`, so
|
||||
the configured pool was **not clamped**; 68.0 GiB of weights locked + 11.0 GiB
|
||||
of KV pool + 16.5 GiB of working memory = 95.4 GiB in all, leaving 14.7 GiB
|
||||
(475 contiguous 2 MiB blocks) for everything else: frank-small on the 3090 and
|
||||
~28 co-tenant containers. The engine printed its co-tenant stall NOTE and
|
||||
logged 4,350 compaction stalls (285 failed) while reserving the pool, so read
|
||||
that as "tight and fragmented", not "out of memory".
|
||||
The serve script validates launcher → built binary → venv runtime → main GGUF →
|
||||
MTP head → PLE payload + manifest + scale, and exits with a pointer message if
|
||||
any is missing. `PLE_CACHE_MIB=4096` is the vendor production-profile default;
|
||||
the 52.4 GiB PLE is mmap'd and paged, not resident.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`.
|
||||
Rollback: [[frank-large-ik-uncensored]].
|
||||
The halogen Quadlet (`dankclaw/frank-large/quadlet/`) was retired to
|
||||
`.AGENTTRASH/` — one owner per endpoint, now the scripts/ trio.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md`.
|
||||
Rollback: [[halogen-flash-server]].
|
||||
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
@@ -327,25 +326,25 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
|
||||
Rollback: [[frank-small-qwen38-gsq-rco]].
|
||||
|
||||
### frank-medium — llama.cpp layer split, MTP drafter (Dankserver, ACTIVE)
|
||||
### frank-medium — llama.cpp layer split, DFlash2 drafter (Dankserver, ACTIVE)
|
||||
|
||||
Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root
|
||||
systemd unit `scripts/frank-medium.service`) serving
|
||||
`ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL**
|
||||
(29.29 GiB) with the model's OWN **MTP head** (`blk.<n>.nextn.*` in the file;
|
||||
the z-lab DFlash2 `Q8_0` sidecar stays on disk and in the fetch script — one
|
||||
`SPEC_TYPE` change brings the faster chain back), in **one process across both
|
||||
R9700s**, `--split-mode layer` at a **35:30 tensor split** — one layer moved
|
||||
to ROCm0 on 2026-09-17 because the cards' PCIe paths are not symmetric:
|
||||
ROCm0 (03:00.0) is CPU-root-port Gen4 x16 (~31.5 GB/s) while ROCm1
|
||||
(0b:00.0) rides the B650 chipset at Gen4 x1 (~1.97 GB/s), and with the
|
||||
drafter, output layers and heavier compute buffers pinned to the
|
||||
last-layers card, the bottlenecked card was running 98% while the x16 card
|
||||
sat at 40%. 36:29 would cross a full-attention layer and fails both the
|
||||
1 GiB floor and the fit slack, so 35:30 is the farthest safe shift at this
|
||||
pool. 393216 ctx over **2 slots** of 196608, batch 6144x512, vision
|
||||
via the BF16 mmproj. DSpark was
|
||||
put up in the drafter's place the same day and reverted: it declares no sliding
|
||||
`orcarouter/Qwen3.8-27B-Uncensored-GGUF` at **Q8_0** (27.0 GiB) with the
|
||||
**z-lab DFlash2 `Q4_K_M` drafter** (1.06 GiB). The target file also carries the
|
||||
model's own MTP head (`blk.<n>.nextn.*`, `nextn_predict_layers=1`), so
|
||||
`draft-mtp` is one `SPEC_TYPE` change away — the drafter file stays on disk. In
|
||||
**one process across both R9700s**, `--split-mode layer` at a **32:33 tensor
|
||||
split** — balanced (operator call 2026-09-19, the most even split 65 layers
|
||||
allow; ROCm1, the last-layers card that also carries the drafter and output
|
||||
layers, holds the extra one). The predecessor's 35:30 skew (2026-09-17) leaned
|
||||
work onto the x16 card because ROCm1 (0b:00.0) rides the B650 chipset at Gen4
|
||||
x1 (~1.97 GB/s) while ROCm0 (03:00.0) is CPU-root-port Gen4 x16 (~31.5 GB/s);
|
||||
the balanced point trades that lean for symmetry and was accepted. 327680 ctx
|
||||
over **2 slots** of 163840 — inside the checkpoint's trained 262144 window, so
|
||||
no rope extrapolation (unlike the predecessor's 393216) — batch 6144x512, f16
|
||||
KV, a 16384 MiB host-RAM prompt cache, vision via the BF16 mmproj. DSpark was
|
||||
tried in the drafter's place on 2026-09-16 and reverted: it declares no sliding
|
||||
window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four
|
||||
layers of the split.
|
||||
|
||||
@@ -706,7 +705,7 @@ Active:
|
||||
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
|
||||
| frank-large (CIRU v4.4.1 Orca, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` |
|
||||
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
|
||||
| frank-medium (llama.cpp + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
@@ -718,6 +717,7 @@ Retired (rollback targets and history):
|
||||
|---------|------|--------|---------|
|
||||
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
|
||||
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
|
||||
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
|
||||
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
|
||||
| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
|
||||
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
||||
|
||||
Reference in new issue
Block a user