dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Switch frank-default-small to Ornith Q6_K

Browse Source
This commit is contained in:
Frank committed 2026-08-25 00:29:14 -07:00
1 parent 3cc99d81cc
commit af6ffeeb64
8 files changed
+115 -60
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/LLM/deployment.md
+10 -10
View File
@@ -17,7 +17,7 @@ repository to a running service.
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankserver | dank@dankserver | upstream llama.cpp (Ornith-1.5-35B-A3B-CRACK Q8_0, native MTP draft, MoE experts offloaded via --override-tensor) | frank-default-small | 7770 |
| Dankserver | dank@dankserver | upstream llama.cpp (Ornith-1.5-35B-A3B-CRACK Q6_K, native MTP draft, selected MoE experts offloaded via --override-tensor) | frank-default-small | 7770 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (Qwen3.8-27B-Uncensored-ROCmFP4 Q6 AGENT, standalone MTP Q4_0 head, 262K ctx / 1 slot) | frank-default-large | 7770 |
## Workflow
@@ -122,17 +122,17 @@ curl -sf http://127.0.0.1:7770/health
### Dankserver — frank-default-small
The active service on Dankserver is upstream llama.cpp (b10121) serving
`Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf` under the alias `frank-default-small`
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small`
on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server`
binary with the MoE expert weights pinned to CPU by an explicit
`--override-tensor` regex (the expert `\.ffn_(up|down|gate|gate_up)_(ch|)exps`
pattern, buffer type `CPU`) and 7 CPU threads, native `draft-mtp`
speculative decoding (bundled MoE MTP head, n_max 4), f16 KV, context
688128 (3 slots of ~229K each, 32k headroom per stream), batch 4096 /
ubatch 2048, `--no-mmap`. The
unit is `scripts/frank-default-small.service`. It launches through
binary with expert tensors from blocks 0 through 36 pinned to CPU by an
explicit `--override-tensor` regex. Expert tensors from blocks 37 through 40
stay on CUDA. The service uses 7 CPU threads, native `draft-mtp`
speculative decoding, f16 KV, context 589824 (3 slots of 196608 each), batch
4096, ubatch 2048, and `--no-mmap`. The unit is
`scripts/frank-default-small.service`. It launches through
`scripts/serve-frank-small.sh`, which reads tunables from
`scripts/frank-default-small.env`. See [[dealign-ornith-crack]].
`scripts/frank-default-small.env`. The chat template comes from
`peculiar-ragdoll/Qwen-Sharp-Chat-Templates`. See [[dealign-ornith-crack]].
### Dankclaw — frank-default-large