The box was upgraded from 32 GB to 64 GB. Two vLLM sizing pages
still stated 30 GiB physical RAM; rocm-smi reports 64 GiB and free
shows 61 GiB visible. Update the hardware tables and the ROCm
overcommit guidance to match.
Also removes the vllm-vulkan backend page, its index links, and the
frank-vk-compat plugin from the working tree. The vLLM service on
Dankclaw is being retired in favor of the CIRU llama.cpp runtime.
The Avesed template only shows a generic format; the Qwen3.5-MoE
model kept emitting <function=get_weather(city=...)> (args baked into
the name). Add a WORKED EXAMPLE block derived from the actual tool
schema (name + parameter keys) to the system prompt.
The Avesed tokenizer_config.json ships no chat_template, so vLLM used its
built-in template, which does not render the <tool_call> instructions;
the model then emitted malformed XML (arguments baked into the function
name) and the qwen3_xml parser could not extract clean calls.
Clients send tool_choice=auto; without these flags vLLM rejects with
HTTP 400. The Avesed chat template renders the Qwen3 XML tool format,
so qwen3_xml is the matching parser.
- frank-default-large: GGML_CUDA_FA_ALL_QUANTS=ON is mandatory in the
FastMTP build; without it q8_0 KV + FA falls back to a slow path and
prefill collapses ~12x (95 -> 1312 t/s at 8K, measured)
- frank-default-small: FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE + pure-python
flash_attn (triton_amd) + stub; without it the vision encoder warmup
OOMs via torch SDPA
- MODEL_FILE -> Qwen3.8-27B-Uncensored-Q6_K.gguf (22.43 GB, no bundled
MTP head; repo ships a standalone draft sidecar)
- SPEC_DRAFT_MODEL -> Qwen3.8-27B-Uncensored-draft-Q8_0.gguf (3.16 GB)
- CACHE_TYPE_K=q8_0, CACHE_TYPE_V=f16 (mismatched, FA retained via the
GGML_CUDA_FA_ALL_QUANTS rebuild of head HIP + worker CUDA at 9b05354ec)
- Split kept at 0.53 for phase 1 (Q6_K frees ~4.7 GiB on the worker at
this split); re-maximize after measuring draft placement
- KV math: K 1088 B/token + V 2048 B/token = 784 MiB per FA layer
- FLM_MODEL=qwythos:9b, CTX 262144, turbo+asr kept
- Add FLM_CONFIG_PATH/FLM_XCLBIN_PATH: the custom tag lives in the user
registry (~/.config/flm/model_list.json) + user xclbins; without them
flm hits the unknown-tag fallback and serves llama3.2:1b silently
- Note FLM 0.9.45 -> 1.0.1 runtime upgrade (pack declares flm_version 1.0.0)
0.54 measured: weight boundary moved to 35 blocks but the KV-per-device
count (round(f x 66), independent of weight boundary) flipped to 9 FA ->
KV 9216 MiB -> OOM by ~740 MiB. The flip sits at f ~0.538, just below
the weight threshold for block 35 -> the 35-block config is unreachable.
0.53 gives 34 blocks (0-33), 8 FA, KV 8192 MiB under every rounding rule
(round/floor/ceil of 0.53x66 = 34.98 -> 34 or 35, never 36), predicted
~1.1 GiB free.
Single full-context slot instead of two half-context slots. The 3090 takes
~33 of 66 layers (up from 26): the VRAM budget frees up because the f16 KV
pool shrinks 401400 -> 262144. 33 layers is the max with a safety margin
(34 -> 244 MiB headroom, 35 -> OOM). Verified math: /tmp/vram-math.py from
live journal buffers.
- README: fix broken 'Also See' placeholder URL -> live Gitea LAN URL
- index.md: expand vllmScripts See Also with the endpoints it serves
- OpenWebUI.md: :7770 backend is the llama.cpp PP head (split to the
3090), managed by vllmScripts; note the FLM NPU :52625 alternative;
fix the broken [[llama.cpp Server]] wikilink
- Gitea.md: hosted repositories table (both projects)
- Dankclaw Stack.md: host-level LLM endpoints in the service inventory
- AGENTS.md: runtime note — LLM endpoints live in vllmScripts, not here
Pre-existing failures (untracked ComfyUI WIP): tests 6, 8, 9, 10.
Docs seams green: ok 19 (compose service docs), ok 20 (index links).
Streaming requests with role=developer get HTTP 200 + error JSON as
first body; pi sees a chunkless stream. Fix: compat.supportsDeveloperRole
false on the model entry. Verified by field bisection + headless pi run.
unattended-upgrades rebooted dankclaw at 15:01; the legacy
podman-compose gitea stack had no systemd units, so rootless podman
did not auto-start the containers. Gitea was down ~1 hour.
- add quadlet units (dankclaw/gitea/quadlet/): gitea-postgres,
gitea-server, gitea-caddy + gitea.network (DNS enabled; the default
rootless podman network has dns_enabled=false)
- add gitea stack Caddyfile (upstream gitea-server:3000) and
.env.example; live password stays in /opt/gitea/.env (0600)
- fix latent name bugs in main stack config: DATABASE_HOST
gitea-postgres, Caddyfile upstream gitea-server
- docs: rewrite Gitea page for the quadlet stack (management, boot
persistence, secrets, gotchas, incident log); fix port 3000 in
stack table; Postgres connection notes
- legacy compose retired on target to /opt/gitea/.deprecated/
- scripts: serve-pp-head/worker, envs, systemd units, pp-test.sh orchestrator, pp-bench.py
- head design verified in b10433 source + live 4B smoke test:
* no --device with --rpc (RPC device silently dropped otherwise)
* default device selection = [RPC0, ROCm0] (HIP registers before Vulkan)
* default split proportional to reported free mem = ~20/80 here, pin 1,1
* -dio required for large model over RPC
- docs: qwen3.8-strix-optimizations (ranked no-quant-downgrade levers,
quant landscape, 2026-08-16 test matrix), PP deployment runbook,
index updates, Dankclaw Improvements corrections (Vulkan0, no power_profile sysfs)
- worker built at 9b05354ec in /home/dank/llama-cpp-rpc worktree (CUDA+RPC)
- second head build build-vulkan-rpc (HIP off) for backend A/B
Frank