Enable DFlash speculative decoding, remove mlock
- Add --spec-type draft-dflash (was missing, DFlash was inactive) - Change --load-mode from mlock to none (RLIMIT_MEMLOCK too small for 2.7GB) - Increase validation max_tokens from 256 to 1024 with complex ML prompt - Baseline: 18.76 tok/s (was 7.62 tok/s without DFlash)
This commit is contained in:
1 parent
1949f0651b
commit
25d57184a6
3 files changed
+5
-4
No files matched your search
@@ -26,6 +26,7 @@ echo ""
|
||||
exec "${SERVER_BIN}" \
|
||||
--hf-repo "${TARGET_REPO}" \
|
||||
--spec-draft-hf "${DRAFT_REPO}" \
|
||||
--spec-type draft-dflash \
|
||||
--spec-draft-n-max "${DRAFT_N_MAX}" \
|
||||
--device ROCm0 \
|
||||
--spec-draft-device ROCm0 \
|
||||
@@ -38,4 +39,4 @@ exec "${SERVER_BIN}" \
|
||||
--port "${PORT}" \
|
||||
--host "0.0.0.0" \
|
||||
--threads 16 \
|
||||
--load-mode mlock
|
||||
--load-mode none
|
||||
Reference in new issue
Block a user