Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
128 lines
4.5 KiB
Markdown
128 lines
4.5 KiB
Markdown
---
|
||
type: reference
|
||
title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
||
description: "Retired era: a mixture-of-experts checkpoint served with CPU expert offload and n-gram drafting on a single card."
|
||
tags:
|
||
- llamacpp
|
||
- cuda
|
||
- dankserver
|
||
- moe
|
||
- ornith
|
||
- mtp
|
||
- retired-endpoint
|
||
timestamp: 2026-08-24
|
||
---
|
||
|
||
# dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
||
|
||
> **Active 2026-08-25.** `frank-zealot` serves this model. It
|
||
> replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121).
|
||
> The Q6_K file was downloaded from the model repository and validated with
|
||
> `llama-gguf` before installation.
|
||
|
||
`frank-zealot` (port 7770) serves
|
||
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is
|
||
a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
|
||
attention architecture, abliterated (uncensored). The GGUF also contains a
|
||
bundled MTP head, but this deployment uses ngram-mod instead.
|
||
|
||
## Model Files
|
||
|
||
| File | Size |
|
||
|------|------|
|
||
| `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` | 27.2 GiB |
|
||
| `mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision) | 0.84 GiB |
|
||
|
||
Store the files under `/home/dank/models/Ornith-1.5-35B-A3B-CRACK/`.
|
||
|
||
## Layer Audit and Expert Offload
|
||
|
||
The Q6_K file contains 41 transformer blocks and 753 tensors. Each block
|
||
contains 630 MiB of expert tensors. Blocks 3, 7, 11, 15, 19, 23, 27, 31,
|
||
35, and 39 contain full attention. Block 40 contains the bundled MTP head.
|
||
|
||
The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for
|
||
ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for
|
||
block 40. The expert portion is 630 MiB in every block.
|
||
|
||
The selected configuration keeps all non-expert tensors on CUDA. It also
|
||
keeps expert tensors from every full-attention block, plus every block from 35
|
||
through 40, on CUDA. The unique GPU expert set is
|
||
`3,7,11,15,19,23,27,31,35,36,37,38,39,40`. Expert tensors from all other
|
||
blocks move to CPU through this explicit override:
|
||
|
||
```
|
||
OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
|
||
OVERRIDE_BUFT=CPU
|
||
```
|
||
|
||
The 16K benchmark found the hybrid policy fastest among the three tested
|
||
placements. A later no-spec/ubatch sweep established that `ubatch=1024` is a
|
||
stable low-memory choice. The current service uses ngram-mod only; MTP is
|
||
intentionally disabled. The full-attention expert set and final blocks are
|
||
kept on CUDA.
|
||
|
||
To list the buffer type names on the host, run a bogus override:
|
||
|
||
```
|
||
llama-server --model <ANY> -c 64 -ot foo=BADTYPE
|
||
```
|
||
|
||
On Dankserver the available buffer types are `CPU` and `CUDA0`.
|
||
|
||
## Speculative Decoding
|
||
|
||
The GGUF contains a bundled MTP head, but this deployment deliberately does
|
||
not use it. The current service enables only `ngram-mod`, with a 48-token
|
||
minimum, 64-token maximum, and 24-token lookup match:
|
||
|
||
```
|
||
--spec-type ngram-mod
|
||
--spec-ngram-mod-n-min 48
|
||
--spec-ngram-mod-n-max 64
|
||
--spec-ngram-mod-n-match 24
|
||
```
|
||
|
||
In the controlled 8K test, no speculation produced approximately 53.7–53.9
|
||
generation tokens per second. MTP was slower for that workload, so it remains
|
||
disabled.
|
||
|
||
## Launch Excerpt
|
||
|
||
```
|
||
llama-server \
|
||
-m Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf \
|
||
--mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
|
||
--chat-template-file chat-template.jinja \
|
||
--spec-type ngram-mod \
|
||
--spec-ngram-mod-n-min 48 \
|
||
--spec-ngram-mod-n-max 64 \
|
||
--spec-ngram-mod-n-match 24 \
|
||
--override-tensor "blk\\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
|
||
-ngl all -c 491520 -np 3 \
|
||
--ubatch-size 1024 \
|
||
--cache-type-k f16 --cache-type-v f16 --no-mmap \
|
||
--temp 1.0 --top-p 0.95 --top-k 20
|
||
```
|
||
|
||
## Known Issues
|
||
|
||
1. The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM
|
||
box with the selected expert offload.
|
||
2. The context pool contains 491520 cells. Three slots receive 163840 cells
|
||
each, which is 96K below the model context per stream.
|
||
3. `--n-cpu-moe` remains supported. This service prefers the explicit tensor
|
||
override because the 16K benchmark found the hybrid placement fastest. It
|
||
preserves every full-attention expert block and blocks 35–40 on CUDA.
|
||
4. MTP is bundled in the GGUF but intentionally disabled; ngram-mod is the
|
||
only active speculative method.
|
||
4. The service uses the Qwen-Sharp template from
|
||
`peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through
|
||
`scripts/chat-template.jinja`.
|
||
|
||
## See Also
|
||
|
||
- [[deployment]] — deploy and update process
|
||
- [[Dankserver System Specs]] — the box
|
||
- [[frank-redo-2026-08-17]] — fleet topology histor
|