Switch frank-default-small to Ornith Q6_K

docs/LLM/dealign-ornith-crack.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
-description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q8_0) served by upstream llama.cpp on the RTX 3090. MoE model with the MTP head bundled; expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small.
+description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with the MTP head bundled; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small.
 tags:
   - llamacpp
   - cuda
@@ -14,12 +14,13 @@ timestamp: 2026-08-24
 
 # dealign Ornith-1.5-35B-A3B CRACK on Dankserver
 
-> **Active 2026-08-24.** `frank-default-small` serves this model. It
-> replaced the HauhauCS Qwen3.6-35B-A3B Q6_K_P service on the same upstream
-> llama.cpp binary (b10121).
+> **Active 2026-08-25.** `frank-default-small` serves this model. It
+> replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121).
+> The Q6_K file was downloaded from the model repository and validated with
+> `llama-gguf` before installation.
 
 `frank-default-small` (port 7770) serves
-`Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf` from `dealignai` on Hugging Face. It is
+`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is
 a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
 attention architecture, abliterated (uncensored). The MTP speculative head
 is **bundled in the GGUF**, so `draft-mtp` auto-detects it from the target
@@ -29,26 +30,36 @@ model.
 
 | File | Size |
 |------|------|
-| `Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf` | 37.2 GB |
-| `mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision) | 0.90 GB |
+| `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` | 27.2 GiB |
+| `mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision) | 0.84 GiB |
 
 Store the files under `/home/dank/models/Ornith-1.5-35B-A3B-CRACK/`.
 
-## Expert Offload — `--override-tensor`, not `--n-cpu-moe`
+## Layer Audit and Expert Offload
 
-The service pins the MoE expert weights to CPU with an **explicit tensor
-override** rather than the layer-counter `--n-cpu-moe`. The override uses
-the same internal regex that `--cpu-moe` / `--n-cpu-moe` expand:
+The Q6_K file contains 41 transformer blocks and 753 tensors. Each block
+contains 630 MiB of expert tensors. Blocks 3, 7, 11, 15, 19, 23, 27, 31,
+35, and 39 contain full attention. Block 40 contains the bundled MTP head.
+
+The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for
+ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for
+block 40. The expert portion is 630 MiB in every block.
+
+The service keeps all non-expert tensors on CUDA. It also keeps expert tensors
+from blocks 37 through 40 on CUDA. Those blocks sit next to the output and
+include the bundled MTP head. Expert tensors from blocks 0 through 36 move to
+CPU through this explicit override:
 
 ```
-OVERRIDE_EXPERT=\.ffn_(up|down|gate|gate_up)_(ch|)exps
+OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
 OVERRIDE_BUFT=CPU
 ```
 
-This is passed as `--override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT"`.
-Only the FFN expert tensors move to CPU; every other tensor stays on the
-24 GB card up to `--gpu-layers`. To list the buffer type names on the host,
-run a bogus override and read the error:
+This places 3.08 GiB of expert weights on CUDA and 22.76 GiB on CPU. The
+selection leaves the output-adjacent blocks and MTP path on CUDA while the
+f16 context cache uses the remaining GPU budget.
+
+To list the buffer type names on the host, run a bogus override:
 
 ```
 llama-server --model <ANY> -c 64 -ot foo=BADTYPE
@@ -72,22 +83,27 @@ proven best. `--spec-draft-n-max 4` is the knee.
 
 ```
 llama-server \
-  -m Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf \
+  -m Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf \
   --mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
   --chat-template-file chat-template.jinja \
   --spec-type draft-mtp \
-  --override-tensor "\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
-  -ngl all -c 688128 -np 3 \
+  --override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
+  -ngl all -c 589824 -np 3 \
   --cache-type-k f16 --cache-type-v f16 --no-mmap \
   --temp 1.0 --top-p 0.95 --top-k 20
 ```
 
 ## Known Issues
 
-1. The 37.2 GB Q8_0 file is large for the 62 GB RAM box; lower `PARALLEL`
-   or switch quant if VRAM or RAM is tight.
-2. `--n-cpu-moe` remains supported; this service deliberately prefers the
-   explicit tensor override so only the expert tensors reach CPU.
+1. The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM
+   box with the selected expert offload.
+2. The context pool contains 589824 cells. Three slots receive 196608 cells
+   each, which is 64K below the model context per stream.
+3. `--n-cpu-moe` remains supported. This service prefers the explicit tensor
+   override because it preserves the final four expert blocks on CUDA.
+4. The service uses the Qwen-Sharp template from
+   `peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through
+   `scripts/chat-template.jinja`.
 
 ## See Also