record performance fixes: FA_ALL_QUANTS build flag (dankserver), RDNA triton FA env (dankclaw)
- frank-default-large: GGML_CUDA_FA_ALL_QUANTS=ON is mandatory in the FastMTP build; without it q8_0 KV + FA falls back to a slow path and prefill collapses ~12x (95 -> 1312 t/s at 8K, measured) - frank-default-small: FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE + pure-python flash_attn (triton_amd) + stub; without it the vision encoder warmup OOMs via torch SDPA
This commit is contained in:
1 parent
27788f6f2c
commit
57264f7faa
4 files changed
+43
-5
No files matched your search
@@ -88,8 +88,13 @@ The HauhauCS FastMTP draft sidecar requires a llama.cpp build from commit
|
||||
`HauhauCS-FastMTP-llama.cpp.patch` (from the HF repo). The build lives at
|
||||
`/home/dank/llama.cpp-fastmtp/build` on Dankserver. Loading the draft with
|
||||
a stock binary fails with `expected 5120, 248320, got 5120, 32768`. Do not
|
||||
rebuild from another commit without re-applying the patch. See the
|
||||
HauhauCS HF repo README for the exact build recipe.
|
||||
rebuild from another commit without re-applying the patch.
|
||||
|
||||
**REQUIRED build flags:** `-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON`.
|
||||
Without `GGML_CUDA_FA_ALL_QUANTS` the q8_0 KV cache + flash attention falls
|
||||
back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s
|
||||
vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README
|
||||
for the base recipe.
|
||||
|
||||
### vLLM ROCm install (Dankclaw)
|
||||
|
||||
@@ -107,6 +112,13 @@ The model cache is the default HF cache
|
||||
`~/.cache/huggingface/token`. The retired CIRU package
|
||||
(`/home/dank/ling3-ciru`) documents a source-build fallback for gfx1151.
|
||||
|
||||
**ROCm flash-attention requirement:** the vision encoder needs the Triton
|
||||
flash-attention path. Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (in
|
||||
`frank-default-small.env`) and keep the pure-Python `flash_attn` package
|
||||
(copied from the CIRU venv) plus a `flash_attn_2_cuda` stub in the venv so
|
||||
`flash_attn.flash_attn_triton_amd` is findable. Without it the mm encoder
|
||||
falls back to torch SDPA and engine init OOMs (256 GiB score matrix).
|
||||
|
||||
### Env var naming rule — HIP runtime collision
|
||||
|
||||
The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`,
|
||||
|
||||
Reference in new issue
Block a user