frank-default-large: full native 262144 context per slot (524288 total)
llama.cpp splits -c across -np slots; to give each of the 2 slots the full 262144 native context, set CTX_SIZE=524288. KV ~32 GiB; fits the 96 GiB unified pool. Update env default, serve-script fallback, and the deployment doc line.
This commit is contained in:
1 parent
819b9d448a
commit
a08be1641f
3 files changed
+12
-9
No files matched your search
@@ -37,7 +37,7 @@ fi
|
||||
: "${ALIAS:=frank-default-large}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=52625}"
|
||||
: "${CTX_SIZE:=262144}"
|
||||
: "${CTX_SIZE:=524288}"
|
||||
: "${PARALLEL:=2}"
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=2048}"
|
||||
|
||||
Reference in new issue
Block a user