record performance fixes: FA_ALL_QUANTS build flag (dankserver), RDNA triton FA env (dankclaw)

- frank-default-large: GGML_CUDA_FA_ALL_QUANTS=ON is mandatory in the
  FastMTP build; without it q8_0 KV + FA falls back to a slow path and
  prefill collapses ~12x (95 -> 1312 t/s at 8K, measured)
- frank-default-small: FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE + pure-python
  flash_attn (triton_amd) + stub; without it the vision encoder warmup
  OOMs via torch SDPA
This commit is contained in:
Frank committed 2026-08-17 16:30:22 -07:00
1 parent 27788f6f2c
commit 57264f7faa
4 files changed
+43 -5

No files matched your search

+8
View File
@@ -38,3 +38,11 @@ GPU_MEMORY_UTILIZATION=0.92
NGRAM_LOOKUP_MAX=5
NGRAM_LOOKUP_MIN=1
NUM_SPEC_TOKENS=5
# RDNA flash-attention for the vision encoder (and general FA paths).
# Without this, the mm encoder falls back to torch SDPA and the warmup
# OOMs (256 GiB score matrix at max_model_len; measured 2026-08-17).
# Requires the flash_attn package with flash_attn_triton_amd findable
# (the pure-Python copy from the CIRU package plus a flash_attn_2_cuda
# stub; see docs/frank-redo-2026-08-17.md).
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE