Tune small model expert placement
This commit is contained in:
1 parent
af6ffeeb64
commit
7a17984dbc
5 files changed
+16
-16
No files matched your search
@@ -124,8 +124,8 @@ curl -sf http://127.0.0.1:7770/health
|
||||
The active service on Dankserver is upstream llama.cpp (b10121) serving
|
||||
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small`
|
||||
on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server`
|
||||
binary with expert tensors from blocks 0 through 36 pinned to CPU by an
|
||||
explicit `--override-tensor` regex. Expert tensors from blocks 37 through 40
|
||||
binary with expert tensors from blocks 0 through 35 pinned to CPU by an
|
||||
explicit `--override-tensor` regex. Expert tensors from blocks 36 through 40
|
||||
stay on CUDA. The service uses 7 CPU threads, native `draft-mtp`
|
||||
speculative decoding, f16 KV, context 589824 (3 slots of 196608 each), batch
|
||||
4096, ubatch 2048, and `--no-mmap`. The unit is
|
||||
|
||||
Reference in new issue
Block a user