feat: frank-default-large 2x 256k context slots (CTX_SIZE=524288, PARALLEL=2)
Total KV: ~12.8 GiB (6.4 GiB per slot × 2). Model is 98.5 GiB mmap'd into 116 GiB SVM-backed unified memory.
This commit is contained in:
1 parent
e376d6a34e
commit
e219fdf939
2 files changed
+9
-8
No files matched your search
@@ -44,8 +44,8 @@ fi
|
||||
: "${ALIAS:=frank-default-large}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7770}"
|
||||
: "${CTX_SIZE:=131072}"
|
||||
: "${PARALLEL:=1}"
|
||||
: "${CTX_SIZE:=524288}"
|
||||
: "${PARALLEL:=2}"
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
: "${UBATCH_SIZE:=4096}"
|
||||
: "${NUM_THREADS:=16}"
|
||||
|
||||
Reference in new issue
Block a user