dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

bench: spec decode sweep + fix serve script ngram-only path

Browse Source 
serve-frank-large.sh: only pass --model-draft when SPEC_TYPE contains
'draft-mtp'. ngram-only configs (ngram-mod, ngram-simple, etc.) do not
need a draft model file — loading one wastes VRAM and may error.

bench-spec-decode.py: new benchmark script. Sweeps 4 configs (none,
ngram-mod, draft-mtp, draft-mtp+ngram-mod), generates 16k tokens per
config at temp 0 from a minimal prompt. Uses HTTP API (llama-bench
cannot target --spec-type). Follows benchmarking policy: stop+mask
the production service, bench, unmask+restart, health check.
This commit is contained in:
Frank committed 2026-08-28 00:04:21 -07:00
1 parent fc75ca6ed8
commit 0ff35e6620
2 files changed
+248 -11
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/serve-frank-large.sh
+16 -11
View File
@@ -92,26 +92,31 @@ if [[ ! -x "$LLAMA_BIN" ]]; then
fi
# --- speculative decoding (MTP draft head) --------------------------------
# The MTP head is loaded as a separate GGUF sidecar via --model-draft.
# The MTP draft head is loaded as a separate GGUF sidecar via --model-draft.
# ngram-only specs (ngram-mod, etc.) do NOT need a draft model file.
# Set SPEC_TYPE=none in the env to disable speculative decoding entirely.
SPEC_ARGS=()
if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
DRAFT_MODEL="$DRAFT_MODEL_DIR/$DRAFT_MODEL_FILE"
if [[ ! -r "$DRAFT_MODEL" ]]; then
echo "missing or unreadable draft model: $DRAFT_MODEL" >&2
exit 1
fi
SPEC_ARGS=(
--spec-type "$SPEC_TYPE"
--model-draft "$DRAFT_MODEL"
--spec-draft-n-max "$SPEC_DRAFT_N_MAX"
--spec-draft-n-min "$SPEC_DRAFT_N_MIN"
--spec-draft-p-min "$SPEC_DRAFT_P_MIN"
--spec-draft-ngl "$SPEC_DRAFT_NGL"
--spec-draft-device "$SPEC_DRAFT_DEVICE"
--spec-draft-type-k "$SPEC_DRAFT_TYPE_K"
--spec-draft-type-v "$SPEC_DRAFT_TYPE_V"
)
if [[ "$SPEC_TYPE" == *draft-mtp* ]]; then
DRAFT_MODEL="$DRAFT_MODEL_DIR/$DRAFT_MODEL_FILE"
if [[ ! -r "$DRAFT_MODEL" ]]; then
echo "missing or unreadable draft model: $DRAFT_MODEL" >&2
exit 1
fi
SPEC_ARGS+=(
--model-draft "$DRAFT_MODEL"
--spec-draft-ngl "$SPEC_DRAFT_NGL"
--spec-draft-device "$SPEC_DRAFT_DEVICE"
--spec-draft-type-k "$SPEC_DRAFT_TYPE_K"
--spec-draft-type-v "$SPEC_DRAFT_TYPE_V"
)
fi
fi
SERVER_DIR="$(cd -- "$(dirname -- "$LLAMA_BIN")" && pwd)"