Revert "Remove KV cache quantization, increase batch/ubatch on frank-default-large"
The Aug 13 change crashed frank-default-large in a restart loop. The service
segfaulted ~15s into model load (SIGSEGV in hipMalloc during load). The
benchmark (frank-default-large-benchmark.md) already proved ubatch 2048 OOMs
(40.9GB prefill buffer) and batch 2048 / ubatch 512 is the fits-and-loads
config on the 96GiB UMA budget. Restore q8_0/q8_0 KV cache (FA pairing) and
batch 2048 / ubatch 512.
Also fix deployment.md: KV is q8_0 K / q8_0 V (not f16 V) since 334b6af.
This commit is contained in:
1 parent
c984a3e4e6
commit
6663af1041
3 files changed
+20
-8
No files matched your search
@@ -33,8 +33,10 @@ fi
|
||||
: "${SPEC_NGRAM_MOD_N_MATCH:=24}"
|
||||
: "${DEVICE:=ROCm0}"
|
||||
: "${N_GPU_LAYERS:=all}"
|
||||
: "${BATCH_SIZE:=8192}"
|
||||
: "${UBATCH_SIZE:=2048}"
|
||||
: "${CACHE_TYPE_K:=q8_0}"
|
||||
: "${CACHE_TYPE_V:=q8_0}"
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
: "${TEMP:=1.0}"
|
||||
: "${TOP_K:=20}"
|
||||
: "${TOP_P:=0.95}"
|
||||
@@ -64,6 +66,8 @@ exec "$LLAMA_SERVER" \
|
||||
--device "$DEVICE" \
|
||||
--gpu-layers "$N_GPU_LAYERS" \
|
||||
--fit off \
|
||||
--cache-type-k "$CACHE_TYPE_K" \
|
||||
--cache-type-v "$CACHE_TYPE_V" \
|
||||
--ctx-size "$CTX_SIZE" \
|
||||
--parallel "$PARALLEL" \
|
||||
--batch-size "$BATCH_SIZE" \
|
||||
|
||||
Reference in new issue
Block a user