serverAppConfigs/serverAppConfigs-docs/LLM/frank-large-ciru-orca.md

---

type: reference title: CIRU v4.4.1 Orca on Dankclaw (frank-large) description: The retired frank-large runbook (2026-09-19 to 2026-09-21, rollback target) — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing. tags:

• llm
• ciru
• dankclaw
• frank-large
• strix-halo
• gfx1151
• qwen4exp
• rocm
• mtp timestamp: 2026-09-19

---

CIRU v4.4.1 Orca on Dankclaw (frank-large)

RETIRED 2026-09-21 (replaced by frank-large-unsloth-qwen38-flash-next; rollback target). It was ACTIVE since 2026-09-19 and replaced halogen-flash-server (see halogen-flash-server), which had replaced ik_llama.cpp (see frank-large-ik-uncensored). It was deployed as a root systemd unit (scripts/frank-large.service, User=dank), port 7770, on the same gfx1151 iGPU the halogen quadlet used. The RTX 3090 it did not touch belongs to frank-small-gemma4-12b.

CIRU is a llama.cpp fork tuned for the Qwen3.8-Flash-Next ("qwen4exp") architecture on AMD gfx1151 (Strix Halo). v4.4.1 carries a QSA window hotfix that is what makes vision and the MTP drafter work on this build; the pre-applied hotfix source ships in the model repo under runtime/v4.4.1/qsa-window-hotfix/. Like every other fleet endpoint it is a llama.cpp runtime — llama-bench is its own harness.

Shape

One root unit, one process, one published port:

half	port	reachable
llama-server (OpenAI front-end + engine, single process)	7770	published as host 7770

Unlike the halogen predecessor (two ports: an unauthenticated engine on 8730 and an OpenAI front-end on 8731), CIRU is a single llama-server process — there is no separate engine port to keep private.

/v1/models reports the loaded model name, Qwen3.8-Flash-CIRU-STRIX-Orca (the launcher sets no --alias). The endpoint serves whatever model name a request names — llama.cpp does not reject an unknown id — so the harness's frank-large label in models.yml works without a rename.

Runtime

A private host build, not a container image. The v4.4.1 source with the QSA window hotfix, built on the host against the private ROCm SDK that scripts/ciru/setup-linux-amd.sh installs into the checkout's own venv:

path	what
/home/dank/GitPackages/ciru-runtime-v4.4.1-qsa-hotfix/	the checkout (source + build + venv)
build-gfx1151-sdk/bin/llama-server	the built binary
.venv-rocm/	the private ROCm SDK venv (~5.7 GB); the binary resolves its libs through RUNPATH
scripts/ciru/run-orca-server.sh	the launcher (exec'd by the serve script)
profiles/strix-halo-production.env	the 36-variable kernel bundle + PLE pager defaults, sourced by the launcher AFTER frank-large.env (every line is ${VAR:-default}, so a value set in frank-large.env wins)

The runtime gate

run-orca-server.sh sources run-server.sh, which exits 2 unless ${repo_root}/runtime/hip/lib/libamdhip64.so and runtime/rocr/lib/ libhsa-runtime64.so exist (unversioned, no suffix). It prepends both to LD_LIBRARY_PATH and ldd-checks that the built libllama-common.so carries the "Qwen4Exp MTP image position" marker and the HIP lib carries the flash_attn_index_mask_clear/set/empty symbols before exec.

The build links the venv through RUNPATH, so those two files are an existence gate, not a real dependency. serve-frank-large.sh satisfies the gate idempotently: it finds the venv's unversioned devel symlinks (.venv-rocm/lib/python*/site-packages/_rocm_sdk_devel/lib/) and ln -sfn-s both into runtime/{hip,rocr}/lib/. The first deploy of this endpoint failed exactly here — the build linked the venv but runtime/ did not exist, so the launcher exited 2 and the unit crash-looped until the serve script created the symlinks.

Payload

scripts/download-frank-large-orca.sh assembles jcbtc/Qwen3.8-Flash-CIRU- STRIX-Orca (HuggingFace, public and ungated) into /home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca/, sha256-verified. The launcher refuses to start without the main GGUF, the MTP head and the PLE sidecar:

file	bytes	role
Qwen3.8-Flash-CIRU-STRIX-Orca.gguf	79,397,818,912	the main model (Orca variant)
mtp/Qwen3.8-Flash-CIRU-STRIX-Orca-MTP-Q8_0.gguf	4,135,893,440	the MTP drafter head
ple/ple.payload.bin	52,429,053,952	the per-layer-embedding (PLE) payload — mmap'd and paged, not resident
ple/ple.manifest.json	—	PLE manifest
ple/ple.scale.bf16	—	PLE scale
vision/mmproj-Qwen3.8-Flash-Orca-F16.mmproj	904,004,320	the F16 vision projector (on disk, ENABLE_VISION=0)

The main GGUF ships as transport parts that the download script assembles and verifies; the PLE is the large one (52.4 GB) and is paged by the engine's own workers, not held in RAM.

Deployed config

Tunables live in scripts/frank-large.env; the launcher owns the flag list (-sm none one iGPU no split, -ngl all, f16 KV, flash attention, cont-batching, the PLE sidecar, the MTP drafter, the production kernel-bundle profile). The departures worth naming:

flag	value	why
CONTEXT_SIZE	262144	the model's FULL trained window (the operator's "full context" call); the GGUF carries no rope scaling
PARALLEL_SLOTS	1	one stream — and the MTP drafter requires exactly one slot in this runtime, so the operator's ask and the drafter's constraint are the same number
ENABLE_MTP	1	the MTP drafter, depth 4 (the Orca variant's own mtp_depth_default)
ENABLE_VISION	0	text-only for now; the F16 projector is on disk, one flip to enable. The fleet's image path is frank-small-gemma4-12b either way
PLE_CACHE_MIB	4096	the PLE pager's hot window kept resident; the vendor production-profile default. The 52.4 GB payload is mmap'd and paged around it
HOST	0.0.0.0	the launcher consumes it for --host; the value doubles as a JIT-cache directory name, which is why 0.0.0.0 is safe (HIP reads HOST bare — see the AGENTS.md note)

TEMP, HOST and GPU_MAX_HEAP_SIZE are the three names to avoid for any new tunable: HIP reads them bare.

MTP

The Multi-Token Prediction drafter is a separate Q8_0 head (not inside the main GGUF), loaded by the launcher. It requires exactly one slot, which is why PARALLEL_SLOTS=1. Depth 4 is the Orca variant's default.

Verified at deploy (2026-09-19): the per-task draft acceptance = 0.65625 (21/32), mean len = 3.62 — the drafter is engaged and accepting, not silently standing down.

PLE

The per-layer-embedding payload (52.4 GB) is the Qwen3.8-Flash-Next architecture's signature: token embeddings live outside the main weight set and are paged on demand by the engine's own workers (GGML_QWEN4EXP_PLE_WORKERS=16, GGML_QWEN4EXP_PLE_IO_WORKERS=32 in the production profile). PLE_CACHE_MIB=4096 is the hot window kept resident; the rest is mmap'd file cache. Do not treat the 52.4 GB as resident memory — free and the dashboards will not show it pinned, and that is correct.

Client contract

The differences from a stock llama.cpp endpoint that bite:

• reasoning_content carries the reasoning trace (the standard OpenAI field). The serve script passes --chat-template-kwargs '{"preserve_thinking":true}' so prior thinking blocks are kept across turns; a request-level chat_template_kwargs still wins.
• reasoning_effort is gated. Qwen3.8's template accepts only xhigh, medium, low and 500s on anything else (not a clamp). The harness folds every level onto that set in models.yml (reasoningEffortMap: minimal/low→low, medium→medium, high/max→xhigh) — subagents run at high, which is why the map exists.
• Text only. ENABLE_VISION=0, so image content parts are refused. The fleet's image path is frank-small-gemma4-12b.
• Sampling. The server has no sampler defaults of its own; the harness applies the per-model recipe from models.yml (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0). A client that stays silent gets the launcher's default.
• tools works — the launcher passes --jinja, and the Qwen3.8 template supplies the tool-call grammar. Verified at deploy: a tool call returns tool_calls.

Deploy and operate

./scripts/deploy.sh dankclaw --service frank-large

The handler pulls the checkout (ff-only), (re)links the unit, and gates on /health. First boot is the slow one (model load ~46 s + graph capture); a warm restart is fast.

sudo systemctl status frank-large            # the root unit
sudo journalctl -u frank-large -n 60         # startup account: runtime check, model load, "listening on http://0.0.0.0:7770", MTP draft loaded
curl -s http://127.0.0.1:7770/health         # ok
curl -s http://127.0.0.1:7770/v1/models      # Qwen3.8-Flash-CIRU-STRIX-Orca

Stop and mask before any GPU experiment (the AGENTS.md bench protocol): sudo systemctl stop frank-large && sudo systemctl mask frank-large. The frank-large serve process is a llama-server and so is frank-small on the same host — never pkill -x llama-server. If you must kill by pattern, use pkill -f 'Qwen3.8-Flash-CIRU-STRIX-Orca' (the model path is unique to this endpoint and does not appear in the deploying shell's own command line).

Troubleshooting

symptom	cause
launcher exits 2: "The v4.4 HIP/ROCr runtime is missing"	runtime/{hip,rocr}/lib/*.so absent — the serve script's ln -sfn did not run, or the venv was moved. Re-run the serve script or rebuild: bash $RUNTIME_DIR/scripts/ciru/setup-linux-amd.sh --install-host-deps
"missing SDK venv runtime under .../.venv-rocm"	the venv's unversioned devel symlinks are gone — rebuild the runtime
"missing target model / missing payload"	a download did not finish — re-run scripts/download-frank-large-orca.sh
loads, /health ok, first completion slow (~45-60 s)	the PLE pager pulling the hot window in from file cache — not a crash, do not restart over it
a subagent 500s naming reasoning_effort	the Qwen3.8 template gate — the harness reasoningEffortMap must fold the level onto {xhigh, medium, low}
free / dashboards do not show 52 GB pinned	correct — the PLE is mmap'd file cache, not resident; only PLE_CACHE_MIB is hot

Retrieval: this is a root unit, so journalctl -u frank-large is the durable record (no --user, no sudo needed for the journal once the unit is readable). There is no container log — there is no container.

Rollback

The predecessor is intact: the halogen payload stays at /home/dank/models/halogen (retired, not deleted), and the halogen quadlet is archived under .AGENTTRASH/. Rolling back is a git revert of this change set plus a deploy — but the root unit and the halogen quadlet both claim port 7770, so the handler's port-free gate covers that race. See halogen-flash-server.

The amd_iommu=off kernel argument is separate and reverts on its own — see dankclaw-iommu-translation-off.

Citations

• jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca — weights, MTP head, PLE payload, and the v4.4.1 QSA window-hotfix source tarball
• CIRU runtime v4.4.1 — scripts/ciru/run-server.sh (runtime gate), profiles/strix-halo-production.env (kernel bundle), docs/BUILD_LINUX.md