record performance fixes: FA_ALL_QUANTS build flag (dankserver), RDNA triton FA env (dankclaw)
- frank-default-large: GGML_CUDA_FA_ALL_QUANTS=ON is mandatory in the FastMTP build; without it q8_0 KV + FA falls back to a slow path and prefill collapses ~12x (95 -> 1312 t/s at 8K, measured) - frank-default-small: FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE + pure-python flash_attn (triton_amd) + stub; without it the vision encoder warmup OOMs via torch SDPA
This commit is contained in:
1 parent
27788f6f2c
commit
57264f7faa
4 files changed
+43
-5
No files matched your search
@@ -15,6 +15,10 @@ MODEL_DIR=/home/dank/models/hauhaucs-qwen38
|
||||
MODEL_FILE=Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-IQ4_XS.gguf
|
||||
DRAFT_FILE=Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-FastMTP-32K.gguf
|
||||
# Patched llama.cpp binary. The FastMTP sidecar needs the patch; see AGENTS.md.
|
||||
# REQUIRED build flags: -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON.
|
||||
# Without FA_ALL_QUANTS the q8_0 KV + flash attention fall back to a slow
|
||||
# path and prefill collapses ~12x (95 t/s vs 1300+ t/s at 8K prompt,
|
||||
# measured 2026-08-17). Do not rebuild without it.
|
||||
LLAMA_BIN=/home/dank/llama.cpp-fastmtp/build/bin/llama-server
|
||||
|
||||
ALIAS=frank-default-large
|
||||
|
||||
@@ -38,3 +38,11 @@ GPU_MEMORY_UTILIZATION=0.92
|
||||
NGRAM_LOOKUP_MAX=5
|
||||
NGRAM_LOOKUP_MIN=1
|
||||
NUM_SPEC_TOKENS=5
|
||||
|
||||
# RDNA flash-attention for the vision encoder (and general FA paths).
|
||||
# Without this, the mm encoder falls back to torch SDPA and the warmup
|
||||
# OOMs (256 GiB score matrix at max_model_len; measured 2026-08-17).
|
||||
# Requires the flash_attn package with flash_attn_triton_amd findable
|
||||
# (the pure-Python copy from the CIRU package plus a flash_attn_2_cuda
|
||||
# stub; see docs/frank-redo-2026-08-17.md).
|
||||
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
|
||||
Reference in new issue
Block a user