Frank
7ba1acdbc5
3 slots x 229376 (262144 - 32768) = 688128 total context. Updates the .env, models.json (contextWindow/maxTokens 229376 per slot), and the index/deployment/ornith runbook references.
96 lines
3.1 KiB
Markdown
96 lines
3.1 KiB
Markdown
---
|
|
type: reference
|
|
title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
|
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q8_0) served by upstream llama.cpp on the RTX 3090. MoE model with the MTP head bundled; expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small.
|
|
tags:
|
|
- llamacpp
|
|
- cuda
|
|
- dankserver
|
|
- moe
|
|
- ornith
|
|
- mtp
|
|
timestamp: 2026-08-24
|
|
---
|
|
|
|
# dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
|
|
|
> **Active 2026-08-24.** `frank-default-small` serves this model. It
|
|
> replaced the HauhauCS Qwen3.6-35B-A3B Q6_K_P service on the same upstream
|
|
> llama.cpp binary (b10121).
|
|
|
|
`frank-default-small` (port 7770) serves
|
|
`Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf` from `dealignai` on Hugging Face. It is
|
|
a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
|
|
attention architecture, abliterated (uncensored). The MTP speculative head
|
|
is **bundled in the GGUF**, so `draft-mtp` auto-detects it from the target
|
|
model.
|
|
|
|
## Model Files
|
|
|
|
| File | Size |
|
|
|------|------|
|
|
| `Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf` | 37.2 GB |
|
|
| `mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision) | 0.90 GB |
|
|
|
|
Store the files under `/home/dank/models/Ornith-1.5-35B-A3B-CRACK/`.
|
|
|
|
## Expert Offload — `--override-tensor`, not `--n-cpu-moe`
|
|
|
|
The service pins the MoE expert weights to CPU with an **explicit tensor
|
|
override** rather than the layer-counter `--n-cpu-moe`. The override uses
|
|
the same internal regex that `--cpu-moe` / `--n-cpu-moe` expand:
|
|
|
|
```
|
|
OVERRIDE_EXPERT=\.ffn_(up|down|gate|gate_up)_(ch|)exps
|
|
OVERRIDE_BUFT=CPU
|
|
```
|
|
|
|
This is passed as `--override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT"`.
|
|
Only the FFN expert tensors move to CPU; every other tensor stays on the
|
|
24 GB card up to `--gpu-layers`. To list the buffer type names on the host,
|
|
run a bogus override and read the error:
|
|
|
|
```
|
|
llama-server --model <ANY> -c 64 -ot foo=BADTYPE
|
|
```
|
|
|
|
On Dankserver the available buffer types are `CPU` and `CUDA0`.
|
|
|
|
## Speculative Decoding
|
|
|
|
MoE model with a bundled MTP head -> native `draft-mtp`. The head is
|
|
auto-detected from the target (b10121 creates the MTP draft context against
|
|
the model file). ngram under MTP hurts on this family; MTP alone is the
|
|
proven best. `--spec-draft-n-max 4` is the knee.
|
|
|
|
```
|
|
--spec-type draft-mtp
|
|
--spec-draft-n-max 4
|
|
```
|
|
|
|
## Launch Excerpt
|
|
|
|
```
|
|
llama-server \
|
|
-m Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf \
|
|
--mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
|
|
--chat-template-file chat-template.jinja \
|
|
--spec-type draft-mtp \
|
|
--override-tensor "\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
|
|
-ngl all -c 688128 -np 3 \
|
|
--cache-type-k f16 --cache-type-v f16 --no-mmap \
|
|
--temp 1.0 --top-p 0.95 --top-k 20
|
|
```
|
|
|
|
## Known Issues
|
|
|
|
1. The 37.2 GB Q8_0 file is large for the 62 GB RAM box; lower `PARALLEL`
|
|
or switch quant if VRAM or RAM is tight.
|
|
2. `--n-cpu-moe` remains supported; this service deliberately prefers the
|
|
explicit tensor override so only the expert tensors reach CPU.
|
|
|
|
## See Also
|
|
|
|
- [[deployment]] — deploy and update process
|
|
- [[Dankserver System Specs]] — the box
|
|
- [[frank-redo-2026-08-17]] — fleet topology history |