Swap LLM fleet to Ornith CRACK (small) and kingjones ROCmFP4 STRIX MTP (large); apply Qwen-Sharp chat template
- frank-default-small (Dankserver): Ornith-1.5-35B-A3B-CRACK Q8_0, bundled native draft-mtp (n_max 4), MoE expert FFN tensors pinned to CPU via an explicit --override-tensor \.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU (replaces --n-cpu-moe), Qwen-Sharp chat template. - frank-default-large (Dankclaw): kingjones Qwen3.8-27B-Uncensored-Q6_0 ROCmFP4 AGENT dense GGUF, standalone mtp-Qwen3.8-27B-Q4_0 draft head (--model-draft, draft-mtp n_max 4), Qwen-Sharp chat template. Drop the retired PromptForge .pfs sidecar lanes. - Track scripts/chat-template.jinja (v22.4.0 Sharp) and apply via --chat-template-file on both services. - Docs: new kingjones-strix-mtp and dealign-ornith-crack runbook pages; update index, deployment, and both system specs; models.json names. Smoke-tested both models on the real hardware on scratch ports before commit: model loads, MTP draft attaches, Sharp template applies, chat completion returns.
This commit is contained in:
1 parent
189934af15
commit
4462aa992c
14 files changed
+827
-156
No files matched your search
@@ -0,0 +1,96 @@
|
||||
---
|
||||
type: reference
|
||||
title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
||||
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q8_0) served by upstream llama.cpp on the RTX 3090. MoE model with the MTP head bundled; expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small.
|
||||
tags:
|
||||
- llamacpp
|
||||
- cuda
|
||||
- dankserver
|
||||
- moe
|
||||
- ornith
|
||||
- mtp
|
||||
timestamp: 2026-08-24
|
||||
---
|
||||
|
||||
# dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
||||
|
||||
> **Active 2026-08-24.** `frank-default-small` serves this model. It
|
||||
> replaced the HauhauCS Qwen3.6-35B-A3B Q6_K_P service on the same upstream
|
||||
> llama.cpp binary (b10121).
|
||||
|
||||
`frank-default-small` (port 7770) serves
|
||||
`Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf` from `dealignai` on Hugging Face. It is
|
||||
a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
|
||||
attention architecture, abliterated (uncensored). The MTP speculative head
|
||||
is **bundled in the GGUF**, so `draft-mtp` auto-detects it from the target
|
||||
model.
|
||||
|
||||
## Model Files
|
||||
|
||||
| File | Size |
|
||||
|------|------|
|
||||
| `Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf` | 37.2 GB |
|
||||
| `mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision) | 0.90 GB |
|
||||
|
||||
Store the files under `/home/dank/models/Ornith-1.5-35B-A3B-CRACK/`.
|
||||
|
||||
## Expert Offload — `--override-tensor`, not `--n-cpu-moe`
|
||||
|
||||
The service pins the MoE expert weights to CPU with an **explicit tensor
|
||||
override** rather than the layer-counter `--n-cpu-moe`. The override uses
|
||||
the same internal regex that `--cpu-moe` / `--n-cpu-moe` expand:
|
||||
|
||||
```
|
||||
OVERRIDE_EXPERT=\.ffn_(up|down|gate|gate_up)_(ch|)exps
|
||||
OVERRIDE_BUFT=CPU
|
||||
```
|
||||
|
||||
This is passed as `--override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT"`.
|
||||
Only the FFN expert tensors move to CPU; every other tensor stays on the
|
||||
24 GB card up to `--gpu-layers`. To list the buffer type names on the host,
|
||||
run a bogus override and read the error:
|
||||
|
||||
```
|
||||
llama-server --model <ANY> -c 64 -ot foo=BADTYPE
|
||||
```
|
||||
|
||||
On Dankserver the available buffer types are `CPU` and `CUDA0`.
|
||||
|
||||
## Speculative Decoding
|
||||
|
||||
MoE model with a bundled MTP head -> native `draft-mtp`. The head is
|
||||
auto-detected from the target (b10121 creates the MTP draft context against
|
||||
the model file). ngram under MTP hurts on this family; MTP alone is the
|
||||
proven best. `--spec-draft-n-max 4` is the knee.
|
||||
|
||||
```
|
||||
--spec-type draft-mtp
|
||||
--spec-draft-n-max 4
|
||||
```
|
||||
|
||||
## Launch Excerpt
|
||||
|
||||
```
|
||||
llama-server \
|
||||
-m Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf \
|
||||
--mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
|
||||
--chat-template-file chat-template.jinja \
|
||||
--spec-type draft-mtp \
|
||||
--override-tensor "\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
|
||||
-ngl all -c 786432 -np 3 \
|
||||
--cache-type-k f16 --cache-type-v f16 --no-mmap \
|
||||
--temp 1.0 --top-p 0.95 --top-k 20
|
||||
```
|
||||
|
||||
## Known Issues
|
||||
|
||||
1. The 37.2 GB Q8_0 file is large for the 62 GB RAM box; lower `PARALLEL`
|
||||
or switch quant if VRAM or RAM is tight.
|
||||
2. `--n-cpu-moe` remains supported; this service deliberately prefers the
|
||||
explicit tensor override so only the expert tensors reach CPU.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[deployment]] — deploy and update process
|
||||
- [[Dankserver System Specs]] — the box
|
||||
- [[frank-redo-2026-08-17]] — fleet topology history
|
||||
Reference in new issue
Block a user