- Restore the pre-2026-08-17 Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive
Q6_K_P + ngram-mod setup as frank-default-small on Dankserver (port 7770,
upstream llama.cpp b10121, MoE experts on CPU via --n-cpu-moe 36).
- Rename the CIRU Qwen3.8-27B ActiveFPX setup from frank-default-small to
frank-default-large (Dankclaw, port 52625); bump to CTX_SIZE=262144
(native max) and PARALLEL=2 (96 GiB unified pool).
- Retire the Qwen3.8-27B IQ4_XS FastMTP frank-default-large (Dankserver) to
.AGENTTRASH.
- deploy.sh: SERVICE[dankserver]=frank-default-small,
SERVICE[dankclaw]=frank-default-large; HEALTH_WAIT[dankserver]=600.
- CTX_SIZE 65536 -> 131072 (user request). KV at 128K is 8 GiB
(64 KiB/token f16); estimated ~43 GiB used of the 64 GiB SVM pool,
~21 GiB headroom.
- REASONING off -> on with REASONING_FORMAT=deepseek, so <think>
reasoning lands in message.reasoning_content (OpenAI-compatible)
instead of being left unparsed in content.
Retire the vLLM service (frank-default-small, port 52625) on Dankclaw
and serve jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge through the CIRU
ROCmFPX llama.cpp runtime instead.
- Build llama-server from ciru-ai/ROCmFPX tag
qwen3.8-activefpx-promptforge-v2.3 (commit 103af8b41) against the
private TheRock ROCm 7.15 SDK in the retired ling3-ciru venv. The
model card requires ROCm 7.15 for the advertised Strix Halo
performance; the system ROCm 7.2.4 selects slower kernels.
CK pinned at fdf4bb7fcc984811cef48ce817d89aac064b984a; generated
ck/config.h + ck/version.h copied into the CK source tree so the
ggml-hip build can include them. LLAMA_BUILD_WEBUI=OFF (HF bucket
llama-ui returns 404; server is consumed via the OpenAI API).
- New scripts/serve-frank-small.sh + frank-default-small.env: 64K ctx,
one slot, f16 KV, context checkpoints, prompt cache, native depth-4
MTP on ROCm0, PromptForge DualView sidecars (FFN + GDN), temp 0.7
top-p 0.95, reasoning off. LD_LIBRARY_PATH puts the 7.15 SDK libs
before any system ROCm.
- Update the systemd unit description (same service name and port).
- deploy.sh: update the dankclaw target comment.
- Docs: new docs/ciru-activefpx-promptforge.md with the build runbook;
update docs/index.md and AGENTS.md topology.
- The vLLM venvs (vllm-venv, vllm-vk-venv) were moved to
/home/dank/.AGENTTRASH/ on the target; the vllm-vulkan plugin and
docs were removed from the repo in an earlier commit.
The Avesed template only shows a generic format; the Qwen3.5-MoE
model kept emitting <function=get_weather(city=...)> (args baked into
the name). Add a WORKED EXAMPLE block derived from the actual tool
schema (name + parameter keys) to the system prompt.
The Avesed tokenizer_config.json ships no chat_template, so vLLM used its
built-in template, which does not render the <tool_call> instructions;
the model then emitted malformed XML (arguments baked into the function
name) and the qwen3_xml parser could not extract clean calls.
Clients send tool_choice=auto; without these flags vLLM rejects with
HTTP 400. The Avesed chat template renders the Qwen3 XML tool format,
so qwen3_xml is the matching parser.
- frank-default-large: GGML_CUDA_FA_ALL_QUANTS=ON is mandatory in the
FastMTP build; without it q8_0 KV + FA falls back to a slow path and
prefill collapses ~12x (95 -> 1312 t/s at 8K, measured)
- frank-default-small: FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE + pure-python
flash_attn (triton_amd) + stub; without it the vision encoder warmup
OOMs via torch SDPA
- FLM_MODEL=qwythos:9b, CTX 262144, turbo+asr kept
- Add FLM_CONFIG_PATH/FLM_XCLBIN_PATH: the custom tag lives in the user
registry (~/.config/flm/model_list.json) + user xclbins; without them
flm hits the unknown-tag fallback and serves llama3.2:1b silently
- Note FLM 0.9.45 -> 1.0.1 runtime upgrade (pack declares flm_version 1.0.0)
The f16/f16 KV pool costs 15640 MiB at 800768 cells. At full 800K the
3090 idled at 23966/24576 MiB and the 1472x1472 image warmup graph
failed cudaMalloc (needed +248 MiB). 700416 cells (175104/slot, 4 slots)
leaves ~2.3 GiB free after warmup. Parallel stays 4.
F16 K removes the flash-attn F16-conversion workspace (MMA_F16 prefill
kernel needs no copy when K is already f16). Trades static KV pool size
for the context-proportional workspace growth implicated in the
cublasSgemm_v2 crash class. VERBOSITY=4 run logs real allocation sizes;
refit CTX_SIZE if the pool does not fit on the 24 GiB 3090.
Fix a live crash: cuBLAS cudaErrorInvalidValue in ggml_cuda_mul_mat_cublas
killed the service during a 145k-token prefill burst at ncmoe35/ub3072.
ncmoe 36 moves one more expert layer to CPU: frees ~0.75 GiB peak VRAM
(grid: peak 8606 vs 9360 MiB at ub3072) for -2% prefill / -2% decode.
Keeps prefill priority (top ubatch tier). Docs synced; noted fallback
GGML_CUDA_FORCE_MMQ=1 if pressure persists.
Prefill is the priority for this workload. Switch from decode-max
(ncmoe 32 / ubatch 1024) to prefill-max:
- n-cpu-moe 35, ubatch 3072: prefill ~1458 t/s (top tier), gen ~45.4 t/s,
~1.2 GiB headroom at the 800k pool.
- Gen is independent of ubatch; ub 3072 reaches best prefill with no
decode cost. Docs synced (benchmark Chosen State, deployment, specs).
- N_CPU_MOE 34→35: 5 expert layers on GPU instead of 6. Frees ~0.4-0.8 GiB
VRAM for OOM safety after high-load sessions OOM'd at 32 and 33.
- Sync stale doc references (deployment.md, Dankserver System Specs.md)
that still claimed 32/8; specs now estimate ~1.8 GiB headroom at 35.
The ROCm HIP runtime (libamdhip64) reads the TEMP environment variable as
its JIT kernel-cache temp directory. Commit db3eb01 added TEMP=1.0 to the
env files as a sampling default; the serve scripts export it via set -a.
HIP then used the path "1.0" for its cache, failed, and segfaulted with a
NULL-refcount unref (offset 0x446153 in libamdhip64) during model tensor
upload (ggml_backend_cuda_buffer_set_tensor). This crashed frank-default-
large on every restart from Aug 13 16:22 onward; the earlier c984a3e
"fix" (KV/batch changes) was unrelated.
Rename the sampling variable TEMP -> TEMPERATURE in both env files and
serve scripts. llama-server still receives --temp from the renamed var, so
sampling defaults are unchanged. The HIP runtime no longer sees a bogus
TEMP value. Affects frank-default-large (dankclaw) and frank-default-small
(dankserver).
- frank-default-large: add TEMP=1.0, TOP_K=20, TOP_P=0.95, MIN_P=0.0
(per HauhauCS Qwen3.6-27B model card, thinking-general mode)
- frank-default-small: add same sampling defaults
(per HauhauCS Qwen3.6-35B-A3B model card)
- frank-default-small: reduce N_CPU_MOE from 33 to 34 (6 expert
layers on GPU instead of 7) to prevent OOM during high-load sessions
- Both serve scripts pass --temp, --top-k, --top-p, --min-p to
llama-server; clients can still override per-request via API
- PARALLEL=4 (800000 total ctx), batch=8192, ubatch=2048
- n-cpu-moe=32 (moved 8 expert layers back to GPU)
- ctx-checkpoints=4 for headroom during larger batch compute
- All other tunables unchanged (ngram-mod, no-mmap, no-kv-offload, q8_0/f16 KV)
Lower each stream to 200000 ctx (pool 600000) to free KV VRAM for the larger
batch. Add --no-mmap so the CPU-resident MoE experts load eagerly into RAM
(avoids decode page faults). Keep KV on GPU. Experts already fully on CPU.
Bigger logical/physical batches for prefill throughput. Enable
--ctx-checkpoints to park attention checkpoints in RAM, holding peak GPU
memory down so the larger ubatch fits the 24GB VRAM alongside 3x262144 KV
without OOM. Weight split unchanged (experts CPU, non-experts GPU).
llama.cpp splits --ctx-size across --parallel slots, so 262144 total gave
87552 per stream. Non-negotiable is 262144 per stream: total pool must be
786432.
llama.cpp service replacing retired vllm-awq-qwen36 on port 7770 with alias
frank-default-small. Wrapper sources frank-default-small.env for tunables:
3 parallel slots at 262144 ctx, draft-dflash n-max 7, KV K=q8_0 V=f16 on GPU,
all MoE experts on CPU (--n-cpu-moe 40), non-expert tensors on GPU, mmproj
vision + jinja. Unit: frank-default-small.service (ExecStart in dankserver
repo clone /home/dank/vLLM/scripts).
Frank