dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: frank-default-large 2x 256k context slots (CTX_SIZE=524288, PARALLEL=2)

Browse Source 
Total KV: ~12.8 GiB (6.4 GiB per slot × 2). Model is 98.5 GiB mmap'd
into 116 GiB SVM-backed unified memory.
This commit is contained in:
Frank
2026-08-26 21:26:39 -07:00
parent e376d6a34e
commit e219fdf939
2 changed files with 9 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-large.env
+7 -6
View File
@@ -16,9 +16,10 @@
# Q4_0_ROCMFP4_STRIX_LEAN quant, 98.5 GiB, 3-part GGUF split. # Q4_0_ROCMFP4_STRIX_LEAN quant, 98.5 GiB, 3-part GGUF split.
# Text-only (no mmproj, no MTP draft head). # Text-only (no mmproj, no MTP draft head).
# #
# Context: 131072 (128k) × 1 slot. f16 KV cache. --fit off because the # Context: 262144 (256k) × 2 slots. f16 KV cache (~6.4 GiB per slot,
# auto-fit code segfaults on this model's hyper-connection/indexer config; # ~12.8 GiB total). --fit off because the auto-fit code segfaults on this
# -ngl 999 places all layers on the SVM-backed GPU pool (116 GiB unified). # model's hyper-connection/indexer config; -ngl 999 places all layers on
# the SVM-backed GPU pool (116 GiB unified).
# #
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP # ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
@@ -45,12 +46,12 @@ ALIAS=frank-default-large
BIND_ADDR=0.0.0.0 BIND_ADDR=0.0.0.0
LISTEN_PORT=7770 LISTEN_PORT=7770
# Serving: 131072 context (128k) × 1 slot. f16 KV, context checkpoints, # Serving: 262144 context (256k) × 2 slots. f16 KV, context checkpoints,
# prompt cache. The model is 98.5 GiB loaded via mmap into the Strix Halo # prompt cache. The model is 98.5 GiB loaded via mmap into the Strix Halo
# unified memory pool (116 GiB SVM-backed). --fit off avoids the auto-fit # unified memory pool (116 GiB SVM-backed). --fit off avoids the auto-fit
# segfault on this model's qwen4exp hyper-connection/indexer configuration. # segfault on this model's qwen4exp hyper-connection/indexer configuration.
CTX_SIZE=131072 CTX_SIZE=524288
PARALLEL=1 PARALLEL=2
BATCH_SIZE=4096 BATCH_SIZE=4096
UBATCH_SIZE=4096 UBATCH_SIZE=4096
NUM_THREADS=16 NUM_THREADS=16
scripts/serve-frank-large.sh
+2 -2
View File
@@ -44,8 +44,8 @@ fi
: "${ALIAS:=frank-default-large}" : "${ALIAS:=frank-default-large}"
: "${BIND_ADDR:=0.0.0.0}" : "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7770}" : "${LISTEN_PORT:=7770}"
: "${CTX_SIZE:=131072}" : "${CTX_SIZE:=524288}"
: "${PARALLEL:=1}" : "${PARALLEL:=2}"
: "${BATCH_SIZE:=4096}" : "${BATCH_SIZE:=4096}"
: "${UBATCH_SIZE:=4096}" : "${UBATCH_SIZE:=4096}"
: "${NUM_THREADS:=16}" : "${NUM_THREADS:=16}"