Add FASTGREEDY mode to serve script: reasoning off + argmax fastpath on for fast-greedy bench

FASTGREEDY=1 flips the serve script to the card's default profile:
- reasoning off (no logit budget overhead)
- LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1 (exact-Q8 GPU argmax)
- deterministic sampler (temp 0, top_p 1, top_k 0, min_p 0)

This mirrors the card's 'fast greedy (default)' benchmark config.
Production (FASTGREEDY=0) unchanged.
This commit is contained in:
Frank committed 2026-08-22 12:54:03 -07:00
1 parent 8b3cd2ff79
commit e0329caef8
2 files changed
+39 -8

No files matched your search

+5
View File
@@ -81,3 +81,8 @@ SPEC_DRAFT_P_SPLIT=0.10
# message.reasoning_content, matching the retired v3 service) with
# --reasoning-budget -1 (unrestricted) and chat_template_kwargs
# preserve_thinking=true (preserve thinking tokens across turns).
#
# For fast-greedy bench: comment out the --reasoning on line in serve script
# and set FASTGREEDY=1 in env (see serve script FASTGREEDY section).
# FASTGREEDY=1 flips to: reasoning off, argmax fastpath=1, sampling defaults.
# FASTGREEDY=0 (default) uses the production values above.