dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Reduce small model context and rebalance GPU layers

Browse Source
This commit is contained in:
Frank
2026-08-25 00:59:41 -07:00
parent 7a17984dbc
commit 339325719e
7 changed files with 38 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/LLM/deployment.md
+3 -3
View File
@@ -124,10 +124,10 @@ curl -sf http://127.0.0.1:7770/health
The active service on Dankserver is upstream llama.cpp (b10121) serving
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small`
on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server`
binary with expert tensors from blocks 0 through 35 pinned to CPU by an
explicit `--override-tensor` regex. Expert tensors from blocks 36 through 40
binary with expert tensors from blocks 0 through 32 pinned to CPU by an
explicit `--override-tensor` regex. Expert tensors from blocks 33 through 40
stay on CUDA. The service uses 7 CPU threads, native `draft-mtp`
speculative decoding, f16 KV, context 589824 (3 slots of 196608 each), batch
speculative decoding, f16 KV, context 491520 (3 slots of 163840 each), batch
4096, ubatch 2048, and `--no-mmap`. The unit is
`scripts/frank-default-small.service`. It launches through
`scripts/serve-frank-small.sh`, which reads tunables from