Enable DFlash speculative decoding, remove mlock

- Add --spec-type draft-dflash (was missing, DFlash was inactive)
- Change --load-mode from mlock to none (RLIMIT_MEMLOCK too small for 2.7GB)
- Increase validation max_tokens from 256 to 1024 with complex ML prompt
- Baseline: 18.76 tok/s (was 7.62 tok/s without DFlash)
This commit is contained in:
Frank committed 2026-08-07 16:10:55 -07:00
1 parent 1949f0651b
commit 25d57184a6
3 files changed
+5 -4

No files matched your search

+2 -2
View File
@@ -5,7 +5,7 @@ set -euo pipefail
PORT="${LLAMA_PORT:-8080}"
HOST="${LLAMA_HOST:-127.0.0.1}"
MAX_TOKENS="${MAX_TOKENS:-256}"
MAX_TOKENS="${MAX_TOKENS:-1024}"
echo "=== llama.cpp Validation Test ==="
echo "Target: http://${HOST}:${PORT}"
@@ -36,7 +36,7 @@ RESPONSE=$(curl -s \
-H "Content-Type: application/json" \
-d "{
\"messages\": [
{\"role\": \"user\", \"content\": \"What is 2+2? Answer in one sentence.\"}
{\"role\": \"user\", \"content\": \"Explain the difference between supervised, unsupervised, and reinforcement learning in machine learning. For each type, provide a concrete real-world example, describe the training process, and discuss the advantages and limitations. Then compare them across three dimensions: data requirements, interpretability of results, and computational cost. Finally, describe a hybrid scenario where combining two of these approaches would outperform using either alone.\"}
],
\"max_tokens\": ${MAX_TOKENS},
\"temperature\": 0.7,