dealign Ornith-1.5-35B-A3B CRACK on Dankserver

Active 2026-08-25. frank-default-small serves this model. It replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121). The Q6_K file was downloaded from the model repository and validated with llama-gguf before installation.

frank-default-small (port 7770) serves Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf from dealignai on Hugging Face. It is a MoE model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) + attention architecture, abliterated (uncensored). The MTP speculative head is bundled in the GGUF, so draft-mtp auto-detects it from the target model.

Model Files

File	Size
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf`	27.2 GiB
`mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision)	0.84 GiB

Store the files under /home/dank/models/Ornith-1.5-35B-A3B-CRACK/.

Layer Audit and Expert Offload

The Q6_K file contains 41 transformer blocks and 753 tensors. Each block contains 630 MiB of expert tensors. Blocks 3, 7, 11, 15, 19, 23, 27, 31, 35, and 39 contain full attention. Block 40 contains the bundled MTP head.

The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for block 40. The expert portion is 630 MiB in every block.

The service keeps all non-expert tensors on CUDA. It also keeps expert tensors from blocks 36 through 40 on CUDA. Those blocks sit next to the output and include the bundled MTP head. Expert tensors from blocks 0 through 35 move to CPU through this explicit override:

OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU

This places 3.69 GiB of expert weights on CUDA and 22.13 GiB on CPU. The selection leaves the output-adjacent blocks and MTP path on CUDA while the f16 context cache uses the remaining GPU budget.

To list the buffer type names on the host, run a bogus override:

llama-server --model <ANY> -c 64 -ot foo=BADTYPE

On Dankserver the available buffer types are CPU and CUDA0.

Speculative Decoding

MoE model with a bundled MTP head -> native draft-mtp. The head is auto-detected from the target (b10121 creates the MTP draft context against the model file). ngram under MTP hurts on this family; MTP alone is the proven best. --spec-draft-n-max 4 is the knee.

--spec-type draft-mtp
--spec-draft-n-max 4

Launch Excerpt

llama-server \
  -m Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf \
  --mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
  --chat-template-file chat-template.jinja \
  --spec-type draft-mtp \
  --override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
  -ngl all -c 589824 -np 3 \
  --cache-type-k f16 --cache-type-v f16 --no-mmap \
  --temp 1.0 --top-p 0.95 --top-k 20

Known Issues

The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM box with the selected expert offload.
The context pool contains 589824 cells. Three slots receive 196608 cells each, which is 64K below the model context per stream.
--n-cpu-moe remains supported. This service prefers the explicit tensor override because it preserves the final five expert blocks on CUDA.
The service uses the Qwen-Sharp template from peculiar-ragdoll/Qwen-Sharp-Chat-Templates through scripts/chat-template.jinja.

4.1 KiB Raw Blame History