Add FASTGREEDY mode to serve script: reasoning off + argmax fastpath on for fast-greedy bench
FASTGREEDY=1 flips the serve script to the card's default profile: - reasoning off (no logit budget overhead) - LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1 (exact-Q8 GPU argmax) - deterministic sampler (temp 0, top_p 1, top_k 0, min_p 0) This mirrors the card's 'fast greedy (default)' benchmark config. Production (FASTGREEDY=0) unchanged.
This commit is contained in:
1 parent
8b3cd2ff79
commit
e0329caef8
2 files changed
+39
-8
No files matched your search
@@ -81,3 +81,8 @@ SPEC_DRAFT_P_SPLIT=0.10
|
||||
# message.reasoning_content, matching the retired v3 service) with
|
||||
# --reasoning-budget -1 (unrestricted) and chat_template_kwargs
|
||||
# preserve_thinking=true (preserve thinking tokens across turns).
|
||||
#
|
||||
# For fast-greedy bench: comment out the --reasoning on line in serve script
|
||||
# and set FASTGREEDY=1 in env (see serve script FASTGREEDY section).
|
||||
# FASTGREEDY=1 flips to: reasoning off, argmax fastpath=1, sampling defaults.
|
||||
# FASTGREEDY=0 (default) uses the production values above.
|
||||
Reference in new issue
Block a user