Local overrides to the pinned runner for production chat:
- --reasoning on --reasoning-format deepseek (thinking to
message.reasoning_content, matching the retired v3 service so the
fleet's reasoning clients work)
- --reasoning-budget -1 (unrestricted thinking)
- --chat-template-kwargs '{"preserve_thinking":"true"}' (preserve
thinking tokens in the conversation history across turns)
- batch 4096 / ubatch 2048 (larger than the runner's 2048/512; unified
memory has headroom)
The Kairic IU4 lane (--kairic-edge) and the MTP draft fast path are
unaffected.
The pinned runner sets LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1, but the
exact-Q8 argmax fast path fails closed on any non-greedy request. The
model card's own chat instructions override sampling per request with
Qwen live values (temp 0.7, top-p 0.8, top-k 20, min-p 0,
presence_penalty 1.5), which the fast path rejects (400). Production
chat forces the fast path to 0; the Kairic IU4 lane (--kairic-edge) and
the MTP draft fast path are unaffected.
Serves jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (four-artifact payload) with
the qualified pinned-runner profile: 262K ctx, 1 slot, b2048/ub512,
t16/tb32, f16 KV, 32 ctx checkpoints, 8 GiB prompt cache with idle-slot
persistence, native MTP4, deterministic sampler, reasoning off.
The serve script replicates the release pinned runner verbatim; the only
differences are safe bind vars (no bare HOST env var; HIP collision rule)
and this machine's split 7.15 SDK LD_LIBRARY_PATH layout.
Rollback: revert this commit and redeploy; v3 model files and the v3
binary are untouched.
Restore the CIRU ROCmFPX setup that DFlash2 replaced:
- Restore jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2 runtime
(native depth-4 MTP, PromptForge DualView, ROCm 7.15)
- Archive the DFlash2 Vulkan deployment files in .AGENTTRASH
- DFlash2 was retired the same day it shipped (2026-08-21) for speed
- Revert docs (deployment.md, index.md, AGENTS.md); mark dflash2 page archived
- Restored scripts are byte-identical to the pre-swap CIRU versions
Replace the retired CIRU ROCmFPX runtime with upstream llama.cpp PR #27342
DFlash2 on the Vulkan backend:
- Target: unsloth Qwen3.8-27B-UD-Q8_K_XL.gguf (31.4 GB, already on disk)
- Drafter: z-lab Qwen3.8-27B-DFlash2-Q4_K_M.gguf (block size 8, top-k 16)
- Runtime: /home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server
(GGML_VULKAN=ON; HIP lacks TOP_K so the stack runs on Vulkan0/RADV)
- spec: --spec-type draft-dflash --spec-draft-n-max 8 (clamps to 7)
Validated 2026-08-21 on scratch port 52626: mean acceptance 5.27,
health + reasoning completion OK.
Archive the retired CIRU .env/.serve script in .AGENTTRASH.
Update deployment.md, index.md, AGENTS.md; add docs/dflash2-large-vulkan.md.
User request 2026-08-19 (second pass): parallel slots degraded
performance. PARALLEL=1, CTX_SIZE=262144 (native max, -c equals the
per-slot context with one slot). Concurrent requests queue on the
single slot; the server never preempts a running generation.
pi models.json contextWindow/maxTokens back to 262144 (outside repo).
Docs updated to match.
User request 2026-08-19: lower per-stream context from 262144 to 200000.
CTX_SIZE is the total; llama.cpp splits it across the 2 parallel slots.
f16 KV drops from ~32 GiB to ~24 GiB, adding OOM headroom.
Docs: sync the 2026-08-18 service swap (frank-default-large on dankclaw,
frank-default-small on dankserver) across AGENTS.md, README.md,
deployment.md, ciru-activefpx-promptforge.md, and index.md. The pi
models.json contextWindow/maxTokens for frank-default-large move
262144 -> 200000 (separate file, outside the repo).
Note that the Pi agent reaches the fleet via ~/.pi/agent/models.json
(outside the repo), that the model ids are stable and referenced by
settings.json, and the two resolution rules (default = provider+model pair;
subagents = unique bare id). Gives the per-slot contextWindow values.
llama.cpp splits -c across -np slots; to give each of the 2 slots the full
262144 native context, set CTX_SIZE=524288. KV ~32 GiB; fits the 96 GiB
unified pool. Update env default, serve-script fallback, and the deployment
doc line.
- Restore the pre-2026-08-17 Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive
Q6_K_P + ngram-mod setup as frank-default-small on Dankserver (port 7770,
upstream llama.cpp b10121, MoE experts on CPU via --n-cpu-moe 36).
- Rename the CIRU Qwen3.8-27B ActiveFPX setup from frank-default-small to
frank-default-large (Dankclaw, port 52625); bump to CTX_SIZE=262144
(native max) and PARALLEL=2 (96 GiB unified pool).
- Retire the Qwen3.8-27B IQ4_XS FastMTP frank-default-large (Dankserver) to
.AGENTTRASH.
- deploy.sh: SERVICE[dankserver]=frank-default-small,
SERVICE[dankclaw]=frank-default-large; HEALTH_WAIT[dankserver]=600.
- CTX_SIZE 65536 -> 131072 (user request). KV at 128K is 8 GiB
(64 KiB/token f16); estimated ~43 GiB used of the 64 GiB SVM pool,
~21 GiB headroom.
- REASONING off -> on with REASONING_FORMAT=deepseek, so <think>
reasoning lands in message.reasoning_content (OpenAI-compatible)
instead of being left unparsed in content.
- Dankclaw System Specs: correct RAM to 64 GiB (61 visible), GPU
budget to the rocm-smi 64 GiB SVM figure, and replace the vLLM
relevance section with the CIRU ROCmFPX runtime facts. Note the
retired vLLM venvs in .AGENTTRASH. Refresh the disk figures.
- deployment.md: update the dankclaw target row (CIRU runtime).
Retire the vLLM service (frank-default-small, port 52625) on Dankclaw
and serve jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge through the CIRU
ROCmFPX llama.cpp runtime instead.
- Build llama-server from ciru-ai/ROCmFPX tag
qwen3.8-activefpx-promptforge-v2.3 (commit 103af8b41) against the
private TheRock ROCm 7.15 SDK in the retired ling3-ciru venv. The
model card requires ROCm 7.15 for the advertised Strix Halo
performance; the system ROCm 7.2.4 selects slower kernels.
CK pinned at fdf4bb7fcc984811cef48ce817d89aac064b984a; generated
ck/config.h + ck/version.h copied into the CK source tree so the
ggml-hip build can include them. LLAMA_BUILD_WEBUI=OFF (HF bucket
llama-ui returns 404; server is consumed via the OpenAI API).
- New scripts/serve-frank-small.sh + frank-default-small.env: 64K ctx,
one slot, f16 KV, context checkpoints, prompt cache, native depth-4
MTP on ROCm0, PromptForge DualView sidecars (FFN + GDN), temp 0.7
top-p 0.95, reasoning off. LD_LIBRARY_PATH puts the 7.15 SDK libs
before any system ROCm.
- Update the systemd unit description (same service name and port).
- deploy.sh: update the dankclaw target comment.
- Docs: new docs/ciru-activefpx-promptforge.md with the build runbook;
update docs/index.md and AGENTS.md topology.
- The vLLM venvs (vllm-venv, vllm-vk-venv) were moved to
/home/dank/.AGENTTRASH/ on the target; the vllm-vulkan plugin and
docs were removed from the repo in an earlier commit.
The box was upgraded from 32 GB to 64 GB. Two vLLM sizing pages
still stated 30 GiB physical RAM; rocm-smi reports 64 GiB and free
shows 61 GiB visible. Update the hardware tables and the ROCm
overcommit guidance to match.
Also removes the vllm-vulkan backend page, its index links, and the
frank-vk-compat plugin from the working tree. The vLLM service on
Dankclaw is being retired in favor of the CIRU llama.cpp runtime.
The Avesed template only shows a generic format; the Qwen3.5-MoE
model kept emitting <function=get_weather(city=...)> (args baked into
the name). Add a WORKED EXAMPLE block derived from the actual tool
schema (name + parameter keys) to the system prompt.
The Avesed tokenizer_config.json ships no chat_template, so vLLM used its
built-in template, which does not render the <tool_call> instructions;
the model then emitted malformed XML (arguments baked into the function
name) and the qwen3_xml parser could not extract clean calls.
Clients send tool_choice=auto; without these flags vLLM rejects with
HTTP 400. The Avesed chat template renders the Qwen3 XML tool format,
so qwen3_xml is the matching parser.
- frank-default-large: GGML_CUDA_FA_ALL_QUANTS=ON is mandatory in the
FastMTP build; without it q8_0 KV + FA falls back to a slow path and
prefill collapses ~12x (95 -> 1312 t/s at 8K, measured)
- frank-default-small: FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE + pure-python
flash_attn (triton_amd) + stub; without it the vision encoder warmup
OOMs via torch SDPA
Frank