Frank
55f87fcdce
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
126 lines
4.6 KiB
Markdown
126 lines
4.6 KiB
Markdown
---
|
||
type: reference
|
||
title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
||
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-zealot.
|
||
tags:
|
||
- llamacpp
|
||
- cuda
|
||
- dankserver
|
||
- moe
|
||
- ornith
|
||
- mtp
|
||
timestamp: 2026-08-24
|
||
---
|
||
|
||
# dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
||
|
||
> **Active 2026-08-25.** `frank-zealot` serves this model. It
|
||
> replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121).
|
||
> The Q6_K file was downloaded from the model repository and validated with
|
||
> `llama-gguf` before installation.
|
||
|
||
`frank-zealot` (port 7770) serves
|
||
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is
|
||
a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
|
||
attention architecture, abliterated (uncensored). The GGUF also contains a
|
||
bundled MTP head, but this deployment uses ngram-mod instead.
|
||
|
||
## Model Files
|
||
|
||
| File | Size |
|
||
|------|------|
|
||
| `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` | 27.2 GiB |
|
||
| `mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision) | 0.84 GiB |
|
||
|
||
Store the files under `/home/dank/models/Ornith-1.5-35B-A3B-CRACK/`.
|
||
|
||
## Layer Audit and Expert Offload
|
||
|
||
The Q6_K file contains 41 transformer blocks and 753 tensors. Each block
|
||
contains 630 MiB of expert tensors. Blocks 3, 7, 11, 15, 19, 23, 27, 31,
|
||
35, and 39 contain full attention. Block 40 contains the bundled MTP head.
|
||
|
||
The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for
|
||
ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for
|
||
block 40. The expert portion is 630 MiB in every block.
|
||
|
||
The selected configuration keeps all non-expert tensors on CUDA. It also
|
||
keeps expert tensors from every full-attention block, plus every block from 35
|
||
through 40, on CUDA. The unique GPU expert set is
|
||
`3,7,11,15,19,23,27,31,35,36,37,38,39,40`. Expert tensors from all other
|
||
blocks move to CPU through this explicit override:
|
||
|
||
```
|
||
OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
|
||
OVERRIDE_BUFT=CPU
|
||
```
|
||
|
||
The 16K benchmark found the hybrid policy fastest among the three tested
|
||
placements. A later no-spec/ubatch sweep established that `ubatch=1024` is a
|
||
stable low-memory choice. The current service uses ngram-mod only; MTP is
|
||
intentionally disabled. The full-attention expert set and final blocks are
|
||
kept on CUDA.
|
||
|
||
To list the buffer type names on the host, run a bogus override:
|
||
|
||
```
|
||
llama-server --model <ANY> -c 64 -ot foo=BADTYPE
|
||
```
|
||
|
||
On Dankserver the available buffer types are `CPU` and `CUDA0`.
|
||
|
||
## Speculative Decoding
|
||
|
||
The GGUF contains a bundled MTP head, but this deployment deliberately does
|
||
not use it. The current service enables only `ngram-mod`, with a 48-token
|
||
minimum, 64-token maximum, and 24-token lookup match:
|
||
|
||
```
|
||
--spec-type ngram-mod
|
||
--spec-ngram-mod-n-min 48
|
||
--spec-ngram-mod-n-max 64
|
||
--spec-ngram-mod-n-match 24
|
||
```
|
||
|
||
In the controlled 8K test, no speculation produced approximately 53.7–53.9
|
||
generation tokens per second. MTP was slower for that workload, so it remains
|
||
disabled.
|
||
|
||
## Launch Excerpt
|
||
|
||
```
|
||
llama-server \
|
||
-m Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf \
|
||
--mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
|
||
--chat-template-file chat-template.jinja \
|
||
--spec-type ngram-mod \
|
||
--spec-ngram-mod-n-min 48 \
|
||
--spec-ngram-mod-n-max 64 \
|
||
--spec-ngram-mod-n-match 24 \
|
||
--override-tensor "blk\\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
|
||
-ngl all -c 491520 -np 3 \
|
||
--ubatch-size 1024 \
|
||
--cache-type-k f16 --cache-type-v f16 --no-mmap \
|
||
--temp 1.0 --top-p 0.95 --top-k 20
|
||
```
|
||
|
||
## Known Issues
|
||
|
||
1. The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM
|
||
box with the selected expert offload.
|
||
2. The context pool contains 491520 cells. Three slots receive 163840 cells
|
||
each, which is 96K below the model context per stream.
|
||
3. `--n-cpu-moe` remains supported. This service prefers the explicit tensor
|
||
override because the 16K benchmark found the hybrid placement fastest. It
|
||
preserves every full-attention expert block and blocks 35–40 on CUDA.
|
||
4. MTP is bundled in the GGUF but intentionally disabled; ngram-mod is the
|
||
only active speculative method.
|
||
4. The service uses the Qwen-Sharp template from
|
||
`peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through
|
||
`scripts/chat-template.jinja`.
|
||
|
||
## See Also
|
||
|
||
- [[deployment]] — deploy and update process
|
||
- [[Dankserver System Specs]] — the box
|
||
- [[frank-redo-2026-08-17]] — fleet topology history |