Revert "Remove KV cache quantization, increase batch/ubatch on frank-default-large"

The Aug 13 change crashed frank-default-large in a restart loop. The service
segfaulted ~15s into model load (SIGSEGV in hipMalloc during load). The
benchmark (frank-default-large-benchmark.md) already proved ubatch 2048 OOMs
(40.9GB prefill buffer) and batch 2048 / ubatch 512 is the fits-and-loads
config on the 96GiB UMA budget. Restore q8_0/q8_0 KV cache (FA pairing) and
batch 2048 / ubatch 512.

Also fix deployment.md: KV is q8_0 K / q8_0 V (not f16 V) since 334b6af.
This commit is contained in:
Frank committed 2026-08-13 17:04:02 -07:00
1 parent c984a3e4e6
commit 6663af1041
3 files changed
+20 -8

No files matched your search

+6 -2
View File
@@ -33,8 +33,10 @@ fi
: "${SPEC_NGRAM_MOD_N_MATCH:=24}"
: "${DEVICE:=ROCm0}"
: "${N_GPU_LAYERS:=all}"
: "${BATCH_SIZE:=8192}"
: "${UBATCH_SIZE:=2048}"
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=q8_0}"
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=512}"
: "${TEMP:=1.0}"
: "${TOP_K:=20}"
: "${TOP_P:=0.95}"
@@ -64,6 +66,8 @@ exec "$LLAMA_SERVER" \
--device "$DEVICE" \
--gpu-layers "$N_GPU_LAYERS" \
--fit off \
--cache-type-k "$CACHE_TYPE_K" \
--cache-type-v "$CACHE_TYPE_V" \
--ctx-size "$CTX_SIZE" \
--parallel "$PARALLEL" \
--batch-size "$BATCH_SIZE" \