Enable DFlash speculative decoding, remove mlock

- Add --spec-type draft-dflash (was missing, DFlash was inactive)
- Change --load-mode from mlock to none (RLIMIT_MEMLOCK too small for 2.7GB)
- Increase validation max_tokens from 256 to 1024 with complex ML prompt
- Baseline: 18.76 tok/s (was 7.62 tok/s without DFlash)
This commit is contained in:
Frank committed 2026-08-07 16:10:55 -07:00
1 parent 1949f0651b
commit 25d57184a6
3 files changed
+5 -4

No files matched your search

+2 -1
View File
@@ -26,6 +26,7 @@ echo ""
exec "${SERVER_BIN}" \
--hf-repo "${TARGET_REPO}" \
--spec-draft-hf "${DRAFT_REPO}" \
--spec-type draft-dflash \
--spec-draft-n-max "${DRAFT_N_MAX}" \
--device ROCm0 \
--spec-draft-device ROCm0 \
@@ -38,4 +39,4 @@ exec "${SERVER_BIN}" \
--port "${PORT}" \
--host "0.0.0.0" \
--threads 16 \
--load-mode mlock
--load-mode none