dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: frank-default-large 2x 256k context slots (CTX_SIZE=524288, PARALLEL=2)

Browse Source 
Total KV: ~12.8 GiB (6.4 GiB per slot × 2). Model is 98.5 GiB mmap'd
into 116 GiB SVM-backed unified memory.
This commit is contained in:
Frank
2026-08-26 21:26:39 -07:00
parent e376d6a34e
commit e219fdf939
2 changed files with 9 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-large.env
+7 -6
View File
@@ -16,9 +16,10 @@
# Q4_0_ROCMFP4_STRIX_LEAN quant, 98.5 GiB, 3-part GGUF split.
# Text-only (no mmproj, no MTP draft head).
#
# Context: 131072 (128k) × 1 slot. f16 KV cache. --fit off because the
# auto-fit code segfaults on this model's hyper-connection/indexer config;
# -ngl 999 places all layers on the SVM-backed GPU pool (116 GiB unified).
# Context: 262144 (256k) × 2 slots. f16 KV cache (~6.4 GiB per slot,
# ~12.8 GiB total). --fit off because the auto-fit code segfaults on this
# model's hyper-connection/indexer config; -ngl 999 places all layers on
# the SVM-backed GPU pool (116 GiB unified).
#
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
@@ -45,12 +46,12 @@ ALIAS=frank-default-large
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# Serving: 131072 context (128k) × 1 slot. f16 KV, context checkpoints,
# Serving: 262144 context (256k) × 2 slots. f16 KV, context checkpoints,
# prompt cache. The model is 98.5 GiB loaded via mmap into the Strix Halo
# unified memory pool (116 GiB SVM-backed). --fit off avoids the auto-fit
# segfault on this model's qwen4exp hyper-connection/indexer configuration.
CTX_SIZE=131072
PARALLEL=1
CTX_SIZE=524288
PARALLEL=2
BATCH_SIZE=4096
UBATCH_SIZE=4096
NUM_THREADS=16