From 57264f7faa7f5c898140f49d34b78bf57f348b12 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 17 Aug 2026 16:30:22 -0700 Subject: [PATCH] record performance fixes: FA_ALL_QUANTS build flag (dankserver), RDNA triton FA env (dankclaw) - frank-default-large: GGML_CUDA_FA_ALL_QUANTS=ON is mandatory in the FastMTP build; without it q8_0 KV + FA falls back to a slow path and prefill collapses ~12x (95 -> 1312 t/s at 8K, measured) - frank-default-small: FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE + pure-python flash_attn (triton_amd) + stub; without it the vision encoder warmup OOMs via torch SDPA --- AGENTS.md | 16 ++++++++++++++-- docs/frank-redo-2026-08-17.md | 20 +++++++++++++++++--- scripts/frank-default-large.env | 4 ++++ scripts/frank-default-small.env | 8 ++++++++ 4 files changed, 43 insertions(+), 5 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7fff800..ace6c12 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -88,8 +88,13 @@ The HauhauCS FastMTP draft sidecar requires a llama.cpp build from commit `HauhauCS-FastMTP-llama.cpp.patch` (from the HF repo). The build lives at `/home/dank/llama.cpp-fastmtp/build` on Dankserver. Loading the draft with a stock binary fails with `expected 5120, 248320, got 5120, 32768`. Do not -rebuild from another commit without re-applying the patch. See the -HauhauCS HF repo README for the exact build recipe. +rebuild from another commit without re-applying the patch. + +**REQUIRED build flags:** `-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON`. +Without `GGML_CUDA_FA_ALL_QUANTS` the q8_0 KV cache + flash attention falls +back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s +vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README +for the base recipe. ### vLLM ROCm install (Dankclaw) @@ -107,6 +112,13 @@ The model cache is the default HF cache `~/.cache/huggingface/token`. The retired CIRU package (`/home/dank/ling3-ciru`) documents a source-build fallback for gfx1151. +**ROCm flash-attention requirement:** the vision encoder needs the Triton +flash-attention path. Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (in +`frank-default-small.env`) and keep the pure-Python `flash_attn` package +(copied from the CIRU venv) plus a `flash_attn_2_cuda` stub in the venv so +`flash_attn.flash_attn_triton_amd` is findable. Without it the mm encoder +falls back to torch SDPA and engine init OOMs (256 GiB score matrix). + ### Env var naming rule — HIP runtime collision The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`, diff --git a/docs/frank-redo-2026-08-17.md b/docs/frank-redo-2026-08-17.md index ba4e48f..be16daf 100644 --- a/docs/frank-redo-2026-08-17.md +++ b/docs/frank-redo-2026-08-17.md @@ -72,9 +72,13 @@ Production choice: **131072 (128K)** — the maximum that keeps about git checkout 4df29be4f4c3673f428170fda944a5b19f743bb8 curl -L -o HauhauCS-FastMTP-llama.cpp.patch git apply HauhauCS-FastMTP-llama.cpp.patch - cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release + cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release \ + -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON cmake --build build --config Release -j$(nproc) ``` + `GGML_CUDA_FA_ALL_QUANTS=ON` is mandatory. Without it, prefill collapses + ~12x (the q8_0 KV + flash attention falls back to a slow path; + measured 95 t/s vs 1300+ t/s at 8K on the 3090). 4. Symlink the unit, reload, enable, restart: ``` sudo ln -sf /home/dank/vLLM/scripts/frank-default-large.service /etc/systemd/system/ @@ -100,13 +104,23 @@ Production choice: **131072 (128K)** — the maximum that keeps about ``` python3 -c "from huggingface_hub import snapshot_download; snapshot_download('Avesed/Qwen3.6-35B-A3B-INT8-W8A8')" ``` -4. Symlink the unit, reload, enable, restart: +4. Install the ROCm wheel set and the flash-attention triton path: + ``` + uv pip install torch --torch-backend=auto + uv pip install "vllm==0.27.1+rocm723" --index-url https://wheels.vllm.ai/rocm/ \ + --extra-index-url https://pypi.org/simple --index-strategy unsafe-best-match --no-build-isolation + cp -r /site-packages/flash_attn* /site-packages/ + # add a flash_attn_2_cuda stub module (ROCm build has no compiled .so) + ``` + Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (env file). Without it the + vision encoder falls back to torch SDPA and engine init OOMs. +5. Symlink the unit, reload, enable, restart: ``` sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/ sudo systemctl daemon-reload && sudo systemctl enable frank-default-small sudo systemctl restart frank-default-small ``` -5. Verify: `curl -sf http://127.0.0.1:52625/health` and a chat completion. +6. Verify: `curl -sf http://127.0.0.1:52625/health` and a chat completion. ## Fallback: vLLM source build for gfx1151 diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index 6b5a402..bbb8f45 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -15,6 +15,10 @@ MODEL_DIR=/home/dank/models/hauhaucs-qwen38 MODEL_FILE=Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-IQ4_XS.gguf DRAFT_FILE=Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-FastMTP-32K.gguf # Patched llama.cpp binary. The FastMTP sidecar needs the patch; see AGENTS.md. +# REQUIRED build flags: -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON. +# Without FA_ALL_QUANTS the q8_0 KV + flash attention fall back to a slow +# path and prefill collapses ~12x (95 t/s vs 1300+ t/s at 8K prompt, +# measured 2026-08-17). Do not rebuild without it. LLAMA_BIN=/home/dank/llama.cpp-fastmtp/build/bin/llama-server ALIAS=frank-default-large diff --git a/scripts/frank-default-small.env b/scripts/frank-default-small.env index 8efd650..e5efc30 100644 --- a/scripts/frank-default-small.env +++ b/scripts/frank-default-small.env @@ -38,3 +38,11 @@ GPU_MEMORY_UTILIZATION=0.92 NGRAM_LOOKUP_MAX=5 NGRAM_LOOKUP_MIN=1 NUM_SPEC_TOKENS=5 + +# RDNA flash-attention for the vision encoder (and general FA paths). +# Without this, the mm encoder falls back to torch SDPA and the warmup +# OOMs (256 GiB score matrix at max_model_len; measured 2026-08-17). +# Requires the flash_attn package with flash_attn_triton_amd findable +# (the pure-Python copy from the CIRU package plus a flash_attn_2_cuda +# stub; see docs/frank-redo-2026-08-17.md). +FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE