dealign Ornith-1.5-35B-A3B CRACK on Dankserver

Active 2026-08-25. frank-default-small serves this model. It replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121). The Q6_K file was downloaded from the model repository and validated with llama-gguf before installation.

frank-default-small (port 7770) serves Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf from dealignai on Hugging Face. It is a MoE model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) + attention architecture, abliterated (uncensored). The MTP speculative head is bundled in the GGUF, so draft-mtp auto-detects it from the target model.

Model Files

File	Size
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf`	27.2 GiB
`mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision)	0.84 GiB

Store the files under /home/dank/models/Ornith-1.5-35B-A3B-CRACK/.

Layer Audit and Expert Offload

The Q6_K file contains 41 transformer blocks and 753 tensors. Each block contains 630 MiB of expert tensors. Blocks 3, 7, 11, 15, 19, 23, 27, 31, 35, and 39 contain full attention. Block 40 contains the bundled MTP head.

The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for block 40. The expert portion is 630 MiB in every block.

The measured hybrid configuration keeps all non-expert tensors on CUDA. It also keeps expert tensors from full-attention blocks 27 and 31, plus the output-adjacent blocks 33 through 40, on CUDA. Block 40 includes the bundled MTP head. Expert tensors from all other blocks move to CPU through this explicit override:

OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU

The measured benchmark compared three policies with a 16K-token prompt: full-attention blocks plus block 40, contiguous top blocks 33–40, and this hybrid. The hybrid was fastest at approximately 33.6 generated tokens per second, versus 33.1 for the contiguous-top policy and 30.3 for the full-attention-only policy. Its first-request prompt throughput was about 2711 tokens per second.

To list the buffer type names on the host, run a bogus override:

llama-server --model <ANY> -c 64 -ot foo=BADTYPE

On Dankserver the available buffer types are CPU and CUDA0.

Speculative Decoding

MoE model with a bundled MTP head -> native draft-mtp. The head is auto-detected from the target (b10121 creates the MTP draft context against the model file). ngram under MTP hurts on this family; MTP alone was previously selected for the production configuration. A later 8K API benchmark found that this MTP path produced lower generation throughput than --spec-type none for that workload, so speculative decoding should be re-evaluated before treating it as optimal. --spec-draft-n-max 4 remains the configured value.

--spec-type draft-mtp
--spec-draft-n-max 4

Launch Excerpt

llama-server \
  -m Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf \
  --mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
  --chat-template-file chat-template.jinja \
  --spec-type draft-mtp \
  --override-tensor "blk\\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
  -ngl all -c 491520 -np 3 \
  --cache-type-k f16 --cache-type-v f16 --no-mmap \
  --temp 1.0 --top-p 0.95 --top-k 20

Known Issues

The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM box with the selected expert offload.
The provisional context pool contains 491520 cells. Three slots receive 163840 cells each, which is 96K below the model context per stream.
--n-cpu-moe remains supported. This service currently prefers the explicit tensor override because the 16K benchmark found this hybrid placement fastest. It preserves the final eight blocks plus full-attention blocks 27 and 31 on CUDA.
The service uses the Qwen-Sharp template from peculiar-ragdoll/Qwen-Sharp-Chat-Templates through scripts/chat-template.jinja.

4.6 KiB Raw Blame History Unescape Escape