Replace the retired CIRU ROCmFPX runtime with upstream llama.cpp PR #27342
DFlash2 on the Vulkan backend:
- Target: unsloth Qwen3.8-27B-UD-Q8_K_XL.gguf (31.4 GB, already on disk)
- Drafter: z-lab Qwen3.8-27B-DFlash2-Q4_K_M.gguf (block size 8, top-k 16)
- Runtime: /home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server
(GGML_VULKAN=ON; HIP lacks TOP_K so the stack runs on Vulkan0/RADV)
- spec: --spec-type draft-dflash --spec-draft-n-max 8 (clamps to 7)
Validated 2026-08-21 on scratch port 52626: mean acceptance 5.27,
health + reasoning completion OK.
Archive the retired CIRU .env/.serve script in .AGENTTRASH.
Update deployment.md, index.md, AGENTS.md; add docs/dflash2-large-vulkan.md.
User request 2026-08-19 (second pass): parallel slots degraded
performance. PARALLEL=1, CTX_SIZE=262144 (native max, -c equals the
per-slot context with one slot). Concurrent requests queue on the
single slot; the server never preempts a running generation.
pi models.json contextWindow/maxTokens back to 262144 (outside repo).
Docs updated to match.
User request 2026-08-19: lower per-stream context from 262144 to 200000.
CTX_SIZE is the total; llama.cpp splits it across the 2 parallel slots.
f16 KV drops from ~32 GiB to ~24 GiB, adding OOM headroom.
Docs: sync the 2026-08-18 service swap (frank-default-large on dankclaw,
frank-default-small on dankserver) across AGENTS.md, README.md,
deployment.md, ciru-activefpx-promptforge.md, and index.md. The pi
models.json contextWindow/maxTokens for frank-default-large move
262144 -> 200000 (separate file, outside the repo).
Note that the Pi agent reaches the fleet via ~/.pi/agent/models.json
(outside the repo), that the model ids are stable and referenced by
settings.json, and the two resolution rules (default = provider+model pair;
subagents = unique bare id). Gives the per-slot contextWindow values.
Retire the vLLM service (frank-default-small, port 52625) on Dankclaw
and serve jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge through the CIRU
ROCmFPX llama.cpp runtime instead.
- Build llama-server from ciru-ai/ROCmFPX tag
qwen3.8-activefpx-promptforge-v2.3 (commit 103af8b41) against the
private TheRock ROCm 7.15 SDK in the retired ling3-ciru venv. The
model card requires ROCm 7.15 for the advertised Strix Halo
performance; the system ROCm 7.2.4 selects slower kernels.
CK pinned at fdf4bb7fcc984811cef48ce817d89aac064b984a; generated
ck/config.h + ck/version.h copied into the CK source tree so the
ggml-hip build can include them. LLAMA_BUILD_WEBUI=OFF (HF bucket
llama-ui returns 404; server is consumed via the OpenAI API).
- New scripts/serve-frank-small.sh + frank-default-small.env: 64K ctx,
one slot, f16 KV, context checkpoints, prompt cache, native depth-4
MTP on ROCm0, PromptForge DualView sidecars (FFN + GDN), temp 0.7
top-p 0.95, reasoning off. LD_LIBRARY_PATH puts the 7.15 SDK libs
before any system ROCm.
- Update the systemd unit description (same service name and port).
- deploy.sh: update the dankclaw target comment.
- Docs: new docs/ciru-activefpx-promptforge.md with the build runbook;
update docs/index.md and AGENTS.md topology.
- The vLLM venvs (vllm-venv, vllm-vk-venv) were moved to
/home/dank/.AGENTTRASH/ on the target; the vllm-vulkan plugin and
docs were removed from the repo in an earlier commit.
- frank-default-large: GGML_CUDA_FA_ALL_QUANTS=ON is mandatory in the
FastMTP build; without it q8_0 KV + FA falls back to a slow path and
prefill collapses ~12x (95 -> 1312 t/s at 8K, measured)
- frank-default-small: FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE + pure-python
flash_attn (triton_amd) + stub; without it the vision encoder warmup
OOMs via torch SDPA
Cold loads after the cache drop measured 17 minutes on 2026-08-13, longer
than the previous 600s wait. HEALTH_WAIT[dankclaw]=600 iterations (20 min),
dankserver stays at 300 (10 min). AGENTS.md: update cold-load figure.
- deployment.md: document the sudoers NOPASSWD restart helpers on both
targets, the exact-argument matching quirk, and the 2026-08-13
'premature job release' investigation (server never preempts; aborts
were client-side; Dankserver crashed twice under researcher load).
- DFlash/DSpark doc: MTP requires MTP tensors in the target GGUF; the
HauhauCS Qwen3.6-27B GGUF has none (verified tensor listing); MTP-only
draft GGUFs (a4lg/lym00) pair via --spec-draft-model; flag rename
--spec-type mtp -> draft-mtp in b9180.
- AGENTS.md: operational quirks for the helpers, journal abort-vs-release
signature, and the Dankserver cublasSgemm crash signature.
Deterministic deployment instead of agent ritual:
- DROP_CACHES per target (dankclaw=1): runs 'sync; echo 3 >
/proc/sys/vm/drop_caches' before systemctl restart. Without it the
model load stalls at 28 GiB VRAM when the model pages are cached.
- FRANK_SUDO_PW env var: remote sudo pipes it via 'sudo -S' so the
script runs headless. Falls back to plain sudo otherwise.
- Health check runs curl on the target over ssh (127.0.0.1:7770 was
previously checked from the workstation, which can never see it).
Retire vllm-awq-qwen36 (AWQ) from target maps, README, deployment, KB index,
Dankserver specs, DFlash and FP8 docs. New dankserver runtime: upstream
llama.cpp, port 7770, alias frank-default-small, 3x262144 ctx, DFlash Q8_0
draft, MoE experts offloaded to CPU, attention+KV on GPU (K q8_0, V f16).
deploy.sh now maps dankserver to frank-default-small and symlinks its unit.
models.json: dankserver baseUrl -> :7770, contextWindow -> 262144.
Update target maps (AGENTS, README, deployment, deploy.sh) from the retired
ROCmFPX llama.cpp stack to the active vLLM-ROCm CIRU INT4 service on port 7770
with MTP speculative decoding. Align Dankserver to the running
vllm-awq-qwen36 (AWQ, port 34401). Refresh KB pages: Dankclaw system specs
(gpu-mem-util 0.95, CIRU venv), quantization choices (INT4 primary, MXFP4
experiment), DFlash/DSpark Dankclaw section, and the KB index topology.
- scripts/deploy.sh: target-aware deploy (dankclaw/dankserver). Pushes main,
pulls on the target, installs/enables systemd unit, restarts service, and
checks the health endpoint; --dry-run preview; pre-flight dirty-tree check
- docs/deployment.md: full deployment/update process with mermaid workflow,
target map, manual update commands, service units, and troubleshooting
- AGENTS.md: add Deployment & Update Process section (deploy.sh usage, target
map, mermaid workflow) and a Summary table row
- docs/index.md: crosslink deployment doc
Frank