frank-default-large: cap CTX_SIZE at 32768 (author quick-start) — 87 GiB fully-resident model + context buffer OOMs HIP heap (50.8 GiB at 131072, 103 GiB at 262144)
This commit is contained in:
1 parent
7ac172c4f4
commit
101b1f8be2
4 files changed
+25
-20
No files matched your search
@@ -45,7 +45,7 @@ fi
|
||||
: "${ALIAS:=frank-default-large}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7770}"
|
||||
: "${CTX_SIZE:=131072}"
|
||||
: "${CTX_SIZE:=32768}"
|
||||
: "${PARALLEL:=1}"
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
: "${UBATCH_SIZE:=4096}"
|
||||
|
||||
Reference in new issue
Block a user