0 Fork 0

Files

T

Frank 4462aa992c Swap LLM fleet to Ornith CRACK (small) and kingjones ROCmFP4 STRIX MTP (large); apply Qwen-Sharp chat template

- frank-default-small (Dankserver): Ornith-1.5-35B-A3B-CRACK Q8_0, bundled
  native draft-mtp (n_max 4), MoE expert FFN tensors pinned to CPU via an
  explicit --override-tensor \.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU
  (replaces --n-cpu-moe), Qwen-Sharp chat template.
- frank-default-large (Dankclaw): kingjones Qwen3.8-27B-Uncensored-Q6_0
  ROCmFP4 AGENT dense GGUF, standalone mtp-Qwen3.8-27B-Q4_0 draft head
  (--model-draft, draft-mtp n_max 4), Qwen-Sharp chat template. Drop the
  retired PromptForge .pfs sidecar lanes.
- Track scripts/chat-template.jinja (v22.4.0 Sharp) and apply via
  --chat-template-file on both services.
- Docs: new kingjones-strix-mtp and dealign-ornith-crack runbook pages;
  update index, deployment, and both system specs; models.json names.

Smoke-tested both models on the real hardware on scratch ports before
commit: model loads, MTP draft attaches, Sharp template applies, chat
completion returns.

2026-08-24 21:35:11 -07:00

3.0 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw

Active 2026-08-24. frank-default-large serves this model. It replaced the jcbtc v3 CIRU ActiveFPX PromptForge payload on the same ROCmFPX runtime and the same CIRU llama.cpp binary. See ciru-activefpx-promptforge for the retired v3 page.

frank-default-large (port 7770) serves Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP from kingjones777 on Hugging Face. It is a dense Qwen3.8-27B, abliterated (uncensored), quantized to the ROCmFP4 Q6 AGENT format (ftype 114). The MTP speculative head ships as a separate Q4_0 draft GGUF, so a bundled-head auto-detect does not apply here; the head is loaded with --model-draft.

Model Files

The card ships six text files plus the vision projector. The service uses the Q6 AGENT file (requested for the fleet), the separate MTP draft head and the Q8 vision projector.

File	Size
`Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf`	23.15 GiB
`mtp-Qwen3.8-27B-Q4_0.gguf` (draft head)	1.56 GiB
`mmproj-Qwen3.8-27B-Q8_0.gguf` (vision)	0.59 GiB

Store the files under /home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/.

Speculative Decoding

Dense model, so the finding measured on the dense Qwen3.8 family holds: MTP alone at --spec-draft-n-max 4 is the knee. ngram under MTP hurts (measured -4 to -5 t/s on the v3; see ciru-activefpx-promptforge). The draft head is a separate file loaded via:

--spec-type draft-mtp
--model-draft mtp-Qwen3.8-27B-Q4_0.gguf
--spec-draft-n-max 4
--spec-draft-n-min 0
--spec-draft-p-min 0.0

The draft head loads on ROCm1 with full offload (--spec-draft-ngl 999) and f16 KV.

Launch Excerpt

The key flags on the CIRU ROCmFPX runtime:

llama-server \
  -m Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf \
  --mmproj mmproj-Qwen3.8-27B-Q8_0.gguf \
  --chat-template-file chat-template.jinja \
  -ngl 999 -dev ROCm0 -c 262144 -np 1 \
  --spec-type draft-mtp --model-draft mtp-Qwen3.8-27B-Q4_0.gguf \
  --spec-draft-ngl 999 --spec-draft-n-max 4 \
  -fa on --no-mmap --cache-type-k f16 --cache-type-v f16

Known Issues

GPU only: Vulkan, CUDA, and CPU cannot load these files.
--spec-draft-n-max defaults to 16; set 4.
reasoning_effort: "none" throws in the Qwen API. Use enable_thinking: false instead.
Small max_tokens with thinking on returns empty content. The Sharp template drives thinking on by default (effort medium).

3.0 KiB Raw Blame History