dealign Ornith-1.5-35B-A3B CRACK on Dankserver

Active 2026-08-25. frank-default-small serves this model. It replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121). The Q6_K file was downloaded from the model repository and validated with llama-gguf before installation.

frank-default-small (port 7770) serves Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf from dealignai on Hugging Face. It is a MoE model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) + attention architecture, abliterated (uncensored). The MTP speculative head is bundled in the GGUF, so draft-mtp auto-detects it from the target model.

Model Files

File	Size
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf`	27.2 GiB
`mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision)	0.84 GiB

Store the files under /home/dank/models/Ornith-1.5-35B-A3B-CRACK/.

Layer Audit and Expert Offload

The Q6_K file contains 41 transformer blocks and 753 tensors. Each block contains 630 MiB of expert tensors. Blocks 3, 7, 11, 15, 19, 23, 27, 31, 35, and 39 contain full attention. Block 40 contains the bundled MTP head.

The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for block 40. The expert portion is 630 MiB in every block.

The provisional service keeps all non-expert tensors on CUDA. It also keeps expert tensors from blocks 33 through 40 on CUDA. Those blocks sit next to the output and include the bundled MTP head. Expert tensors from blocks 0 through 32 move to CPU through this explicit override:

OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU

The previous five-block configuration placed 3.69 GiB of expert weights on CUDA and 22.13 GiB on CPU. This provisional eight-block configuration adds approximately 1.89 GiB of expert weights to CUDA while reducing the context pool by 98304 cells. The final placement will be selected after measuring this new headroom and discussing layer priorities.

To list the buffer type names on the host, run a bogus override:

llama-server --model <ANY> -c 64 -ot foo=BADTYPE

On Dankserver the available buffer types are CPU and CUDA0.

Speculative Decoding

MoE model with a bundled MTP head -> native draft-mtp. The head is auto-detected from the target (b10121 creates the MTP draft context against the model file). ngram under MTP hurts on this family; MTP alone is the proven best. --spec-draft-n-max 4 is the knee.

--spec-type draft-mtp
--spec-draft-n-max 4

Launch Excerpt

llama-server \
  -m Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf \
  --mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
  --chat-template-file chat-template.jinja \
  --spec-type draft-mtp \
  --override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
  -ngl all -c 491520 -np 3 \
  --cache-type-k f16 --cache-type-v f16 --no-mmap \
  --temp 1.0 --top-p 0.95 --top-k 20

Known Issues

The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM box with the selected expert offload.
The provisional context pool contains 491520 cells. Three slots receive 163840 cells each, which is 96K below the model context per stream.
--n-cpu-moe remains supported. This service currently prefers the explicit tensor override because it preserves the final eight expert blocks on CUDA pending the next layer-priority decision.
The service uses the Qwen-Sharp template from peculiar-ragdoll/Qwen-Sharp-Chat-Templates through scripts/chat-template.jinja.

4.3 KiB Raw Blame History