Files

T

Frank 8212d4451a docs: rename frank-default-large/small/sightsound to frank-large/small/npu across vault

Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.

2026-09-03 16:17:14 -07:00

4.6 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

dealign Ornith-1.5-35B-A3B CRACK on Dankserver

Active 2026-08-25. frank-small serves this model. It replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121). The Q6_K file was downloaded from the model repository and validated with llama-gguf before installation.

frank-small (port 7770) serves Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf from dealignai on Hugging Face. It is a MoE model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) + attention architecture, abliterated (uncensored). The GGUF also contains a bundled MTP head, but this deployment uses ngram-mod instead.

Model Files

File	Size
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf`	27.2 GiB
`mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision)	0.84 GiB

Store the files under /home/dank/models/Ornith-1.5-35B-A3B-CRACK/.

Layer Audit and Expert Offload

The Q6_K file contains 41 transformer blocks and 753 tensors. Each block contains 630 MiB of expert tensors. Blocks 3, 7, 11, 15, 19, 23, 27, 31, 35, and 39 contain full attention. Block 40 contains the bundled MTP head.

The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for block 40. The expert portion is 630 MiB in every block.

The selected configuration keeps all non-expert tensors on CUDA. It also keeps expert tensors from every full-attention block, plus every block from 35 through 40, on CUDA. The unique GPU expert set is 3,7,11,15,19,23,27,31,35,36,37,38,39,40. Expert tensors from all other blocks move to CPU through this explicit override:

OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU

The 16K benchmark found the hybrid policy fastest among the three tested placements. A later no-spec/ubatch sweep established that ubatch=1024 is a stable low-memory choice. The current service uses ngram-mod only; MTP is intentionally disabled. The full-attention expert set and final blocks are kept on CUDA.

To list the buffer type names on the host, run a bogus override:

llama-server --model <ANY> -c 64 -ot foo=BADTYPE

On Dankserver the available buffer types are CPU and CUDA0.

Speculative Decoding

The GGUF contains a bundled MTP head, but this deployment deliberately does not use it. The current service enables only ngram-mod, with a 48-token minimum, 64-token maximum, and 24-token lookup match:

--spec-type ngram-mod
--spec-ngram-mod-n-min 48
--spec-ngram-mod-n-max 64
--spec-ngram-mod-n-match 24

In the controlled 8K test, no speculation produced approximately 53.7–53.9 generation tokens per second. MTP was slower for that workload, so it remains disabled.

Launch Excerpt

llama-server \
  -m Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf \
  --mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
  --chat-template-file chat-template.jinja \
  --spec-type ngram-mod \
  --spec-ngram-mod-n-min 48 \
  --spec-ngram-mod-n-max 64 \
  --spec-ngram-mod-n-match 24 \
  --override-tensor "blk\\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
  -ngl all -c 491520 -np 3 \
  --ubatch-size 1024 \
  --cache-type-k f16 --cache-type-v f16 --no-mmap \
  --temp 1.0 --top-p 0.95 --top-k 20

Known Issues

The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM box with the selected expert offload.
The context pool contains 491520 cells. Three slots receive 163840 cells each, which is 96K below the model context per stream.
--n-cpu-moe remains supported. This service prefers the explicit tensor override because the 16K benchmark found the hybrid placement fastest. It preserves every full-attention expert block and blocks 35–40 on CUDA.
MTP is bundled in the GGUF but intentionally disabled; ngram-mod is the only active speculative method.
The service uses the Qwen-Sharp template from peculiar-ragdoll/Qwen-Sharp-Chat-Templates through scripts/chat-template.jinja.

4.6 KiB Raw Blame History Unescape Escape