112 lines
4.1 KiB
Markdown
112 lines
4.1 KiB
Markdown
---
|
|
type: reference
|
|
title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
|
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with the MTP head bundled; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small.
|
|
tags:
|
|
- llamacpp
|
|
- cuda
|
|
- dankserver
|
|
- moe
|
|
- ornith
|
|
- mtp
|
|
timestamp: 2026-08-24
|
|
---
|
|
|
|
# dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
|
|
|
> **Active 2026-08-25.** `frank-default-small` serves this model. It
|
|
> replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121).
|
|
> The Q6_K file was downloaded from the model repository and validated with
|
|
> `llama-gguf` before installation.
|
|
|
|
`frank-default-small` (port 7770) serves
|
|
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is
|
|
a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
|
|
attention architecture, abliterated (uncensored). The MTP speculative head
|
|
is **bundled in the GGUF**, so `draft-mtp` auto-detects it from the target
|
|
model.
|
|
|
|
## Model Files
|
|
|
|
| File | Size |
|
|
|------|------|
|
|
| `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` | 27.2 GiB |
|
|
| `mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision) | 0.84 GiB |
|
|
|
|
Store the files under `/home/dank/models/Ornith-1.5-35B-A3B-CRACK/`.
|
|
|
|
## Layer Audit and Expert Offload
|
|
|
|
The Q6_K file contains 41 transformer blocks and 753 tensors. Each block
|
|
contains 630 MiB of expert tensors. Blocks 3, 7, 11, 15, 19, 23, 27, 31,
|
|
35, and 39 contain full attention. Block 40 contains the bundled MTP head.
|
|
|
|
The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for
|
|
ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for
|
|
block 40. The expert portion is 630 MiB in every block.
|
|
|
|
The service keeps all non-expert tensors on CUDA. It also keeps expert tensors
|
|
from blocks 36 through 40 on CUDA. Those blocks sit next to the output and
|
|
include the bundled MTP head. Expert tensors from blocks 0 through 35 move to
|
|
CPU through this explicit override:
|
|
|
|
```
|
|
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
|
|
OVERRIDE_BUFT=CPU
|
|
```
|
|
|
|
This places 3.69 GiB of expert weights on CUDA and 22.13 GiB on CPU. The
|
|
selection leaves the output-adjacent blocks and MTP path on CUDA while the
|
|
f16 context cache uses the remaining GPU budget.
|
|
|
|
To list the buffer type names on the host, run a bogus override:
|
|
|
|
```
|
|
llama-server --model <ANY> -c 64 -ot foo=BADTYPE
|
|
```
|
|
|
|
On Dankserver the available buffer types are `CPU` and `CUDA0`.
|
|
|
|
## Speculative Decoding
|
|
|
|
MoE model with a bundled MTP head -> native `draft-mtp`. The head is
|
|
auto-detected from the target (b10121 creates the MTP draft context against
|
|
the model file). ngram under MTP hurts on this family; MTP alone is the
|
|
proven best. `--spec-draft-n-max 4` is the knee.
|
|
|
|
```
|
|
--spec-type draft-mtp
|
|
--spec-draft-n-max 4
|
|
```
|
|
|
|
## Launch Excerpt
|
|
|
|
```
|
|
llama-server \
|
|
-m Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf \
|
|
--mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
|
|
--chat-template-file chat-template.jinja \
|
|
--spec-type draft-mtp \
|
|
--override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
|
|
-ngl all -c 589824 -np 3 \
|
|
--cache-type-k f16 --cache-type-v f16 --no-mmap \
|
|
--temp 1.0 --top-p 0.95 --top-k 20
|
|
```
|
|
|
|
## Known Issues
|
|
|
|
1. The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM
|
|
box with the selected expert offload.
|
|
2. The context pool contains 589824 cells. Three slots receive 196608 cells
|
|
each, which is 64K below the model context per stream.
|
|
3. `--n-cpu-moe` remains supported. This service prefers the explicit tensor
|
|
override because it preserves the final five expert blocks on CUDA.
|
|
4. The service uses the Qwen-Sharp template from
|
|
`peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through
|
|
`scripts/chat-template.jinja`.
|
|
|
|
## See Also
|
|
|
|
- [[deployment]] — deploy and update process
|
|
- [[Dankserver System Specs]] — the box
|
|
- [[frank-redo-2026-08-17]] — fleet topology history |