Swap LLM fleet to Ornith CRACK (small) and kingjones ROCmFP4 STRIX MTP (large); apply Qwen-Sharp chat template

- frank-default-small (Dankserver): Ornith-1.5-35B-A3B-CRACK Q8_0, bundled
  native draft-mtp (n_max 4), MoE expert FFN tensors pinned to CPU via an
  explicit --override-tensor \.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU
  (replaces --n-cpu-moe), Qwen-Sharp chat template.
- frank-default-large (Dankclaw): kingjones Qwen3.8-27B-Uncensored-Q6_0
  ROCmFP4 AGENT dense GGUF, standalone mtp-Qwen3.8-27B-Q4_0 draft head
  (--model-draft, draft-mtp n_max 4), Qwen-Sharp chat template. Drop the
  retired PromptForge .pfs sidecar lanes.
- Track scripts/chat-template.jinja (v22.4.0 Sharp) and apply via
  --chat-template-file on both services.
- Docs: new kingjones-strix-mtp and dealign-ornith-crack runbook pages;
  update index, deployment, and both system specs; models.json names.

Smoke-tested both models on the real hardware on scratch ports before
commit: model loads, MTP draft attaches, Sharp template applies, chat
completion returns.
This commit is contained in:
Frank committed 2026-08-24 21:35:11 -07:00
1 parent 189934af15
commit 4462aa992c
14 files changed
+827 -156

No files matched your search

+96
View File
@@ -0,0 +1,96 @@
---
type: reference
title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q8_0) served by upstream llama.cpp on the RTX 3090. MoE model with the MTP head bundled; expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small.
tags:
- llamacpp
- cuda
- dankserver
- moe
- ornith
- mtp
timestamp: 2026-08-24
---
# dealign Ornith-1.5-35B-A3B CRACK on Dankserver
> **Active 2026-08-24.** `frank-default-small` serves this model. It
> replaced the HauhauCS Qwen3.6-35B-A3B Q6_K_P service on the same upstream
> llama.cpp binary (b10121).
`frank-default-small` (port 7770) serves
`Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf` from `dealignai` on Hugging Face. It is
a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
attention architecture, abliterated (uncensored). The MTP speculative head
is **bundled in the GGUF**, so `draft-mtp` auto-detects it from the target
model.
## Model Files
| File | Size |
|------|------|
| `Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf` | 37.2 GB |
| `mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision) | 0.90 GB |
Store the files under `/home/dank/models/Ornith-1.5-35B-A3B-CRACK/`.
## Expert Offload — `--override-tensor`, not `--n-cpu-moe`
The service pins the MoE expert weights to CPU with an **explicit tensor
override** rather than the layer-counter `--n-cpu-moe`. The override uses
the same internal regex that `--cpu-moe` / `--n-cpu-moe` expand:
```
OVERRIDE_EXPERT=\.ffn_(up|down|gate|gate_up)_(ch|)exps
OVERRIDE_BUFT=CPU
```
This is passed as `--override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT"`.
Only the FFN expert tensors move to CPU; every other tensor stays on the
24 GB card up to `--gpu-layers`. To list the buffer type names on the host,
run a bogus override and read the error:
```
llama-server --model <ANY> -c 64 -ot foo=BADTYPE
```
On Dankserver the available buffer types are `CPU` and `CUDA0`.
## Speculative Decoding
MoE model with a bundled MTP head -> native `draft-mtp`. The head is
auto-detected from the target (b10121 creates the MTP draft context against
the model file). ngram under MTP hurts on this family; MTP alone is the
proven best. `--spec-draft-n-max 4` is the knee.
```
--spec-type draft-mtp
--spec-draft-n-max 4
```
## Launch Excerpt
```
llama-server \
-m Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf \
--mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
--chat-template-file chat-template.jinja \
--spec-type draft-mtp \
--override-tensor "\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
-ngl all -c 786432 -np 3 \
--cache-type-k f16 --cache-type-v f16 --no-mmap \
--temp 1.0 --top-p 0.95 --top-k 20
```
## Known Issues
1. The 37.2 GB Q8_0 file is large for the 62 GB RAM box; lower `PARALLEL`
or switch quant if VRAM or RAM is tight.
2. `--n-cpu-moe` remains supported; this service deliberately prefers the
explicit tensor override so only the expert tensors reach CPU.
## See Also
- [[deployment]] — deploy and update process
- [[Dankserver System Specs]] — the box
- [[frank-redo-2026-08-17]] — fleet topology history