record performance fixes: FA_ALL_QUANTS build flag (dankserver), RDNA triton FA env (dankclaw)

- frank-default-large: GGML_CUDA_FA_ALL_QUANTS=ON is mandatory in the
  FastMTP build; without it q8_0 KV + FA falls back to a slow path and
  prefill collapses ~12x (95 -> 1312 t/s at 8K, measured)
- frank-default-small: FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE + pure-python
  flash_attn (triton_amd) + stub; without it the vision encoder warmup
  OOMs via torch SDPA
This commit is contained in:
Frank committed 2026-08-17 16:30:22 -07:00
1 parent 27788f6f2c
commit 57264f7faa
4 files changed
+43 -5

No files matched your search

+14 -2
View File
@@ -88,8 +88,13 @@ The HauhauCS FastMTP draft sidecar requires a llama.cpp build from commit
`HauhauCS-FastMTP-llama.cpp.patch` (from the HF repo). The build lives at
`/home/dank/llama.cpp-fastmtp/build` on Dankserver. Loading the draft with
a stock binary fails with `expected 5120, 248320, got 5120, 32768`. Do not
rebuild from another commit without re-applying the patch. See the
HauhauCS HF repo README for the exact build recipe.
rebuild from another commit without re-applying the patch.
**REQUIRED build flags:** `-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON`.
Without `GGML_CUDA_FA_ALL_QUANTS` the q8_0 KV cache + flash attention falls
back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s
vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README
for the base recipe.
### vLLM ROCm install (Dankclaw)
@@ -107,6 +112,13 @@ The model cache is the default HF cache
`~/.cache/huggingface/token`. The retired CIRU package
(`/home/dank/ling3-ciru`) documents a source-build fallback for gfx1151.
**ROCm flash-attention requirement:** the vision encoder needs the Triton
flash-attention path. Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (in
`frank-default-small.env`) and keep the pure-Python `flash_attn` package
(copied from the CIRU venv) plus a `flash_attn_2_cuda` stub in the venv so
`flash_attn.flash_attn_triton_amd` is findable. Without it the mm encoder
falls back to torch SDPA and engine init OOMs (256 GiB score matrix).
### Env var naming rule — HIP runtime collision
The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`,