122 lines
4.6 KiB
Markdown
122 lines
4.6 KiB
Markdown
---
|
||
type: reference
|
||
title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
||
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with the MTP head bundled; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small.
|
||
tags:
|
||
- llamacpp
|
||
- cuda
|
||
- dankserver
|
||
- moe
|
||
- ornith
|
||
- mtp
|
||
timestamp: 2026-08-24
|
||
---
|
||
|
||
# dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
||
|
||
> **Active 2026-08-25.** `frank-default-small` serves this model. It
|
||
> replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121).
|
||
> The Q6_K file was downloaded from the model repository and validated with
|
||
> `llama-gguf` before installation.
|
||
|
||
`frank-default-small` (port 7770) serves
|
||
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is
|
||
a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
|
||
attention architecture, abliterated (uncensored). The MTP speculative head
|
||
is **bundled in the GGUF**, so `draft-mtp` auto-detects it from the target
|
||
model.
|
||
|
||
## Model Files
|
||
|
||
| File | Size |
|
||
|------|------|
|
||
| `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` | 27.2 GiB |
|
||
| `mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision) | 0.84 GiB |
|
||
|
||
Store the files under `/home/dank/models/Ornith-1.5-35B-A3B-CRACK/`.
|
||
|
||
## Layer Audit and Expert Offload
|
||
|
||
The Q6_K file contains 41 transformer blocks and 753 tensors. Each block
|
||
contains 630 MiB of expert tensors. Blocks 3, 7, 11, 15, 19, 23, 27, 31,
|
||
35, and 39 contain full attention. Block 40 contains the bundled MTP head.
|
||
|
||
The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for
|
||
ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for
|
||
block 40. The expert portion is 630 MiB in every block.
|
||
|
||
The measured hybrid configuration keeps all non-expert tensors on CUDA. It
|
||
also keeps expert tensors from full-attention blocks 27 and 31, plus the
|
||
output-adjacent blocks 33 through 40, on CUDA. Block 40 includes the bundled
|
||
MTP head. Expert tensors from all other blocks move to CPU through this
|
||
explicit override:
|
||
|
||
```
|
||
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
|
||
OVERRIDE_BUFT=CPU
|
||
```
|
||
|
||
The measured benchmark compared three policies with a 16K-token prompt:
|
||
full-attention blocks plus block 40, contiguous top blocks 33–40, and this
|
||
hybrid. The hybrid was fastest at approximately 33.6 generated tokens per
|
||
second, versus 33.1 for the contiguous-top policy and 30.3 for the
|
||
full-attention-only policy. Its first-request prompt throughput was about
|
||
2711 tokens per second.
|
||
|
||
To list the buffer type names on the host, run a bogus override:
|
||
|
||
```
|
||
llama-server --model <ANY> -c 64 -ot foo=BADTYPE
|
||
```
|
||
|
||
On Dankserver the available buffer types are `CPU` and `CUDA0`.
|
||
|
||
## Speculative Decoding
|
||
|
||
MoE model with a bundled MTP head -> native `draft-mtp`. The head is
|
||
auto-detected from the target (b10121 creates the MTP draft context against
|
||
the model file). ngram under MTP hurts on this family; MTP alone was
|
||
previously selected for the production configuration. A later 8K API
|
||
benchmark found that this MTP path produced lower generation throughput than
|
||
`--spec-type none` for that workload, so speculative decoding should be
|
||
re-evaluated before treating it as optimal. `--spec-draft-n-max 4` remains the
|
||
configured value.
|
||
|
||
```
|
||
--spec-type draft-mtp
|
||
--spec-draft-n-max 4
|
||
```
|
||
|
||
## Launch Excerpt
|
||
|
||
```
|
||
llama-server \
|
||
-m Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf \
|
||
--mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
|
||
--chat-template-file chat-template.jinja \
|
||
--spec-type draft-mtp \
|
||
--override-tensor "blk\\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
|
||
-ngl all -c 491520 -np 3 \
|
||
--cache-type-k f16 --cache-type-v f16 --no-mmap \
|
||
--temp 1.0 --top-p 0.95 --top-k 20
|
||
```
|
||
|
||
## Known Issues
|
||
|
||
1. The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM
|
||
box with the selected expert offload.
|
||
2. The provisional context pool contains 491520 cells. Three slots receive
|
||
163840 cells each, which is 96K below the model context per stream.
|
||
3. `--n-cpu-moe` remains supported. This service currently prefers the
|
||
explicit tensor override because the 16K benchmark found this hybrid
|
||
placement fastest. It preserves the final eight blocks plus full-attention
|
||
blocks 27 and 31 on CUDA.
|
||
4. The service uses the Qwen-Sharp template from
|
||
`peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through
|
||
`scripts/chat-template.jinja`.
|
||
|
||
## See Also
|
||
|
||
- [[deployment]] — deploy and update process
|
||
- [[Dankserver System Specs]] — the box
|
||
- [[frank-redo-2026-08-17]] — fleet topology history |