- CTX_SIZE 65536 -> 131072 (user request). KV at 128K is 8 GiB
(64 KiB/token f16); estimated ~43 GiB used of the 64 GiB SVM pool,
~21 GiB headroom.
- REASONING off -> on with REASONING_FORMAT=deepseek, so <think>
reasoning lands in message.reasoning_content (OpenAI-compatible)
instead of being left unparsed in content.
- Dankclaw System Specs: correct RAM to 64 GiB (61 visible), GPU
budget to the rocm-smi 64 GiB SVM figure, and replace the vLLM
relevance section with the CIRU ROCmFPX runtime facts. Note the
retired vLLM venvs in .AGENTTRASH. Refresh the disk figures.
- deployment.md: update the dankclaw target row (CIRU runtime).
Retire the vLLM service (frank-default-small, port 52625) on Dankclaw
and serve jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge through the CIRU
ROCmFPX llama.cpp runtime instead.
- Build llama-server from ciru-ai/ROCmFPX tag
qwen3.8-activefpx-promptforge-v2.3 (commit 103af8b41) against the
private TheRock ROCm 7.15 SDK in the retired ling3-ciru venv. The
model card requires ROCm 7.15 for the advertised Strix Halo
performance; the system ROCm 7.2.4 selects slower kernels.
CK pinned at fdf4bb7fcc984811cef48ce817d89aac064b984a; generated
ck/config.h + ck/version.h copied into the CK source tree so the
ggml-hip build can include them. LLAMA_BUILD_WEBUI=OFF (HF bucket
llama-ui returns 404; server is consumed via the OpenAI API).
- New scripts/serve-frank-small.sh + frank-default-small.env: 64K ctx,
one slot, f16 KV, context checkpoints, prompt cache, native depth-4
MTP on ROCm0, PromptForge DualView sidecars (FFN + GDN), temp 0.7
top-p 0.95, reasoning off. LD_LIBRARY_PATH puts the 7.15 SDK libs
before any system ROCm.
- Update the systemd unit description (same service name and port).
- deploy.sh: update the dankclaw target comment.
- Docs: new docs/ciru-activefpx-promptforge.md with the build runbook;
update docs/index.md and AGENTS.md topology.
- The vLLM venvs (vllm-venv, vllm-vk-venv) were moved to
/home/dank/.AGENTTRASH/ on the target; the vllm-vulkan plugin and
docs were removed from the repo in an earlier commit.
The box was upgraded from 32 GB to 64 GB. Two vLLM sizing pages
still stated 30 GiB physical RAM; rocm-smi reports 64 GiB and free
shows 61 GiB visible. Update the hardware tables and the ROCm
overcommit guidance to match.
Also removes the vllm-vulkan backend page, its index links, and the
frank-vk-compat plugin from the working tree. The vLLM service on
Dankclaw is being retired in favor of the CIRU llama.cpp runtime.
The Avesed template only shows a generic format; the Qwen3.5-MoE
model kept emitting <function=get_weather(city=...)> (args baked into
the name). Add a WORKED EXAMPLE block derived from the actual tool
schema (name + parameter keys) to the system prompt.
The Avesed tokenizer_config.json ships no chat_template, so vLLM used its
built-in template, which does not render the <tool_call> instructions;
the model then emitted malformed XML (arguments baked into the function
name) and the qwen3_xml parser could not extract clean calls.
Clients send tool_choice=auto; without these flags vLLM rejects with
HTTP 400. The Avesed chat template renders the Qwen3 XML tool format,
so qwen3_xml is the matching parser.
- frank-default-large: GGML_CUDA_FA_ALL_QUANTS=ON is mandatory in the
FastMTP build; without it q8_0 KV + FA falls back to a slow path and
prefill collapses ~12x (95 -> 1312 t/s at 8K, measured)
- frank-default-small: FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE + pure-python
flash_attn (triton_amd) + stub; without it the vision encoder warmup
OOMs via torch SDPA
- MODEL_FILE -> Qwen3.8-27B-Uncensored-Q6_K.gguf (22.43 GB, no bundled
MTP head; repo ships a standalone draft sidecar)
- SPEC_DRAFT_MODEL -> Qwen3.8-27B-Uncensored-draft-Q8_0.gguf (3.16 GB)
- CACHE_TYPE_K=q8_0, CACHE_TYPE_V=f16 (mismatched, FA retained via the
GGML_CUDA_FA_ALL_QUANTS rebuild of head HIP + worker CUDA at 9b05354ec)
- Split kept at 0.53 for phase 1 (Q6_K frees ~4.7 GiB on the worker at
this split); re-maximize after measuring draft placement
- KV math: K 1088 B/token + V 2048 B/token = 784 MiB per FA layer
- FLM_MODEL=qwythos:9b, CTX 262144, turbo+asr kept
- Add FLM_CONFIG_PATH/FLM_XCLBIN_PATH: the custom tag lives in the user
registry (~/.config/flm/model_list.json) + user xclbins; without them
flm hits the unknown-tag fallback and serves llama3.2:1b silently
- Note FLM 0.9.45 -> 1.0.1 runtime upgrade (pack declares flm_version 1.0.0)
0.54 measured: weight boundary moved to 35 blocks but the KV-per-device
count (round(f x 66), independent of weight boundary) flipped to 9 FA ->
KV 9216 MiB -> OOM by ~740 MiB. The flip sits at f ~0.538, just below
the weight threshold for block 35 -> the 35-block config is unreachable.
0.53 gives 34 blocks (0-33), 8 FA, KV 8192 MiB under every rounding rule
(round/floor/ceil of 0.53x66 = 34.98 -> 34 or 35, never 36), predicted
~1.1 GiB free.
Single full-context slot instead of two half-context slots. The 3090 takes
~33 of 66 layers (up from 26): the VRAM budget frees up because the f16 KV
pool shrinks 401400 -> 262144. 33 layers is the max with a safety margin
(34 -> 244 MiB headroom, 35 -> OOM). Verified math: /tmp/vram-math.py from
live journal buffers.
Streaming requests with role=developer get HTTP 200 + error JSON as
first body; pi sees a chunkless stream. Fix: compat.supportsDeveloperRole
false on the model entry. Verified by field bisection + headless pi run.
- scripts: serve-pp-head/worker, envs, systemd units, pp-test.sh orchestrator, pp-bench.py
- head design verified in b10433 source + live 4B smoke test:
* no --device with --rpc (RPC device silently dropped otherwise)
* default device selection = [RPC0, ROCm0] (HIP registers before Vulkan)
* default split proportional to reported free mem = ~20/80 here, pin 1,1
* -dio required for large model over RPC
- docs: qwen3.8-strix-optimizations (ranked no-quant-downgrade levers,
quant landscape, 2026-08-16 test matrix), PP deployment runbook,
index updates, Dankclaw Improvements corrections (Vulkan0, no power_profile sysfs)
- worker built at 9b05354ec in /home/dank/llama-cpp-rpc worktree (CUDA+RPC)
- second head build build-vulkan-rpc (HIP off) for backend A/B
Recording the 54-config matrix, the realistic-prompt correction (n_max=5 not
10), the ubatch findings (256 is the safe/fast point), and the premature-stop
sampling-variance investigation.
Realistic-prompt tuning: n_max=5 peaks ~13.3 t/s (baseline 7.2). matrix used a
repetitive prompt that inflated MTP. n_max=10 is only ~8.6 t/s on real writing.
The f16/f16 KV pool costs 15640 MiB at 800768 cells. At full 800K the
3090 idled at 23966/24576 MiB and the 1472x1472 image warmup graph
failed cudaMalloc (needed +248 MiB). 700416 cells (175104/slot, 4 slots)
leaves ~2.3 GiB free after warmup. Parallel stays 4.
F16 K removes the flash-attn F16-conversion workspace (MMA_F16 prefill
kernel needs no copy when K is already f16). Trades static KV pool size
for the context-proportional workspace growth implicated in the
cublasSgemm_v2 crash class. VERBOSITY=4 run logs real allocation sizes;
refit CTX_SIZE if the pool does not fit on the 24 GiB 3090.
Fix a live crash: cuBLAS cudaErrorInvalidValue in ggml_cuda_mul_mat_cublas
killed the service during a 145k-token prefill burst at ncmoe35/ub3072.
ncmoe 36 moves one more expert layer to CPU: frees ~0.75 GiB peak VRAM
(grid: peak 8606 vs 9360 MiB at ub3072) for -2% prefill / -2% decode.
Keeps prefill priority (top ubatch tier). Docs synced; noted fallback
GGML_CUDA_FORCE_MMQ=1 if pressure persists.
Compiles the 5-subagent research fan-out (llama.cpp RPC, vLLM PP,
bandwidth math, mixed-vendor experiments, Qwen3.6-27B spec methods) into
the KB under docs/Pipeline Parallelism/ with an index, expected
performance estimates (1/2.5/10 GbE), and an EDUCATIONAL-ONLY two-node
Qwen3.6-27B Q8 split script set (MTP/DFlash/ngram-mod variants, DSpark
noted as vLLM-only). Nothing here is deployed or intended to run.
Prefill is the priority for this workload. Switch from decode-max
(ncmoe 32 / ubatch 1024) to prefill-max:
- n-cpu-moe 35, ubatch 3072: prefill ~1458 t/s (top tier), gen ~45.4 t/s,
~1.2 GiB headroom at the 800k pool.
- Gen is independent of ubatch; ub 3072 reaches best prefill with no
decode cost. Docs synced (benchmark Chosen State, deployment, specs).
Retest MTP at n_max 2 (vs 3) for the acceptance/speed tradeoff, keeping
MTP-alone + ubatch 8192. Re-benches gen t/s and re-runs the strict
parallel-safety test.
MTP+ngram benched 12.0-13.6 t/s (vs MTP alone 15.3-19.5) and is
parallel-safe, but slower - layering does not pay off. Back to MTP
alone. Test ubatch 8192 (== batch) using the VRAM headroom; watch for
OOM during load.
Test the MTP+ngram layering (ngram re-drafts on top of the MTP draft) at
ubatch 4096. Must clear the strict parallel-safety test and beat MTP
alone (15.3-19.5 t/s) to stay.
MTP (draft-mtp, n_max 3, a4lg MTP-ONLY-Q8_0) benches 15.3-19.5 t/s vs
ngram-mod 6.7-8.7 t/s and passes the strict LCP-reuse parallel test
(sim 0.996 / f_keep 0.994 launch while another slot generates - the
pattern that killed DFlash). ubatch 4096 to use the ~33 GiB VRAM
headroom; watch peak during load.
ngram-mod benchmark + parallel-safety test passed (A and B both survive
the LCP-reuse launch that killed DFlash). Now benching MTP sidecar
(a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf, draft-mtp, n_max 3).
DFlash causes premature EOS on a generating slot when another task lands
with high LCP prompt reuse (shared draft ctx corruption, reproduced
2026-08-14 3x). Benching ngram-mod vs MTP sidecar to pick the
replacement. serve script now passes --spec-draft-model only for
draft-* spec types.
Cold loads after the cache drop measured 17 minutes on 2026-08-13, longer
than the previous 600s wait. HEALTH_WAIT[dankclaw]=600 iterations (20 min),
dankserver stays at 300 (10 min). AGENTS.md: update cold-load figure.
- deployment.md: document the sudoers NOPASSWD restart helpers on both
targets, the exact-argument matching quirk, and the 2026-08-13
'premature job release' investigation (server never preempts; aborts
were client-side; Dankserver crashed twice under researcher load).
- DFlash/DSpark doc: MTP requires MTP tensors in the target GGUF; the
HauhauCS Qwen3.6-27B GGUF has none (verified tensor listing); MTP-only
draft GGUFs (a4lg/lym00) pair via --spec-draft-model; flag rename
--spec-type mtp -> draft-mtp in b9180.
- AGENTS.md: operational quirks for the helpers, journal abort-vs-release
signature, and the Dankserver cublasSgemm crash signature.
The helper scripts (/usr/local/sbin/frank-restart-<large|small>.sh)
cover unit symlink, cache drop (dankclaw), daemon-reload, and restart
in one root-owned script, run via sudoers NOPASSWD. deploy.sh now uses
the helper when present and falls back to inline sudo commands.
- N_CPU_MOE 34→35: 5 expert layers on GPU instead of 6. Frees ~0.4-0.8 GiB
VRAM for OOM safety after high-load sessions OOM'd at 32 and 33.
- Sync stale doc references (deployment.md, Dankserver System Specs.md)
that still claimed 32/8; specs now estimate ~1.8 GiB headroom at 35.
Deterministic deployment instead of agent ritual:
- DROP_CACHES per target (dankclaw=1): runs 'sync; echo 3 >
/proc/sys/vm/drop_caches' before systemctl restart. Without it the
model load stalls at 28 GiB VRAM when the model pages are cached.
- FRANK_SUDO_PW env var: remote sudo pipes it via 'sudo -S' so the
script runs headless. Falls back to plain sudo otherwise.
- Health check runs curl on the target over ssh (127.0.0.1:7770 was
previously checked from the workstation, which can never see it).
Empirically verified on 2026-08-13 (caches dropped): batch 8192 / ubatch
2048 loads and serves at ctx 602100 with 61 GiB VRAM peak. The benchmark
doc's 'ubatch 2048 OOMs' results were confounded by page-cache pressure -
with the model's pages fully cached the KV/graph allocation stalls at
28 GiB VRAM. Drop caches (sync; echo 3 > /proc/sys/vm/drop_caches) before
starting a load to make restarts reliable.
The ROCm HIP runtime (libamdhip64) reads the TEMP environment variable as
its JIT kernel-cache temp directory. Commit db3eb01 added TEMP=1.0 to the
env files as a sampling default; the serve scripts export it via set -a.
HIP then used the path "1.0" for its cache, failed, and segfaulted with a
NULL-refcount unref (offset 0x446153 in libamdhip64) during model tensor
upload (ggml_backend_cuda_buffer_set_tensor). This crashed frank-default-
large on every restart from Aug 13 16:22 onward; the earlier c984a3e
"fix" (KV/batch changes) was unrelated.
Rename the sampling variable TEMP -> TEMPERATURE in both env files and
serve scripts. llama-server still receives --temp from the renamed var, so
sampling defaults are unchanged. The HIP runtime no longer sees a bogus
TEMP value. Affects frank-default-large (dankclaw) and frank-default-small
(dankserver).
The Aug 13 change crashed frank-default-large in a restart loop. The service
segfaulted ~15s into model load (SIGSEGV in hipMalloc during load). The
benchmark (frank-default-large-benchmark.md) already proved ubatch 2048 OOMs
(40.9GB prefill buffer) and batch 2048 / ubatch 512 is the fits-and-loads
config on the 96GiB UMA budget. Restore q8_0/q8_0 KV cache (FA pairing) and
batch 2048 / ubatch 512.
Also fix deployment.md: KV is q8_0 K / q8_0 V (not f16 V) since 334b6af.
- Remove CACHE_TYPE_K/CACHE_TYPE_V (fp16 default on GPU)
- Increase BATCH_SIZE from 2048 to 8192
- Increase UBATCH_SIZE from 512 to 2048
- Remove --cache-type-k and --cache-type-v from llama-server flags
Frank