record performance fixes: FA_ALL_QUANTS build flag (dankserver), RDNA triton FA env (dankclaw)

- frank-default-large: GGML_CUDA_FA_ALL_QUANTS=ON is mandatory in the
  FastMTP build; without it q8_0 KV + FA falls back to a slow path and
  prefill collapses ~12x (95 -> 1312 t/s at 8K, measured)
- frank-default-small: FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE + pure-python
  flash_attn (triton_amd) + stub; without it the vision encoder warmup
  OOMs via torch SDPA

AGENTS.md

@@ -88,8 +88,13 @@ The HauhauCS FastMTP draft sidecar requires a llama.cpp build from commit
 `HauhauCS-FastMTP-llama.cpp.patch` (from the HF repo). The build lives at
 `/home/dank/llama.cpp-fastmtp/build` on Dankserver. Loading the draft with
 a stock binary fails with `expected 5120, 248320, got 5120, 32768`. Do not
-rebuild from another commit without re-applying the patch. See the
+rebuild from another commit without re-applying the patch.
-HauhauCS HF repo README for the exact build recipe.
+
+**REQUIRED build flags:** `-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON`.
+Without `GGML_CUDA_FA_ALL_QUANTS` the q8_0 KV cache + flash attention falls
+back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s
+vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README
+for the base recipe.
 
 ### vLLM ROCm install (Dankclaw)
 
@@ -107,6 +112,13 @@ The model cache is the default HF cache
 `~/.cache/huggingface/token`. The retired CIRU package
 (`/home/dank/ling3-ciru`) documents a source-build fallback for gfx1151.
 
+**ROCm flash-attention requirement:** the vision encoder needs the Triton
+flash-attention path. Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (in
+`frank-default-small.env`) and keep the pure-Python `flash_attn` package
+(copied from the CIRU venv) plus a `flash_attn_2_cuda` stub in the venv so
+`flash_attn.flash_attn_triton_amd` is findable. Without it the mm encoder
+falls back to torch SDPA and engine init OOMs (256 GiB score matrix).
+
 ### Env var naming rule — HIP runtime collision
 
 The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`,

docs/frank-redo-2026-08-17.md

@@ -72,9 +72,13 @@ Production choice: **131072 (128K)** — the maximum that keeps about
    git checkout 4df29be4f4c3673f428170fda944a5b19f743bb8
    curl -L -o HauhauCS-FastMTP-llama.cpp.patch <HF patch URL>
    git apply HauhauCS-FastMTP-llama.cpp.patch
-   cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
+   cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release \
+     -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON
    cmake --build build --config Release -j$(nproc)
    ```
+   `GGML_CUDA_FA_ALL_QUANTS=ON` is mandatory. Without it, prefill collapses
+   ~12x (the q8_0 KV + flash attention falls back to a slow path;
+   measured 95 t/s vs 1300+ t/s at 8K on the 3090).
 4. Symlink the unit, reload, enable, restart:
    ```
    sudo ln -sf /home/dank/vLLM/scripts/frank-default-large.service /etc/systemd/system/
@@ -100,13 +104,23 @@ Production choice: **131072 (128K)** — the maximum that keeps about
    ```
    python3 -c "from huggingface_hub import snapshot_download; snapshot_download('Avesed/Qwen3.6-35B-A3B-INT8-W8A8')"
    ```
-4. Symlink the unit, reload, enable, restart:
+4. Install the ROCm wheel set and the flash-attention triton path:
+   ```
+   uv pip install torch --torch-backend=auto
+   uv pip install "vllm==0.27.1+rocm723" --index-url https://wheels.vllm.ai/rocm/ \
+     --extra-index-url https://pypi.org/simple --index-strategy unsafe-best-match --no-build-isolation
+   cp -r <ciru-venv>/site-packages/flash_attn* <vllm-venv>/site-packages/
+   # add a flash_attn_2_cuda stub module (ROCm build has no compiled .so)
+   ```
+   Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (env file). Without it the
+   vision encoder falls back to torch SDPA and engine init OOMs.
+5. Symlink the unit, reload, enable, restart:
    ```
    sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/
    sudo systemctl daemon-reload && sudo systemctl enable frank-default-small
    sudo systemctl restart frank-default-small
    ```
-5. Verify: `curl -sf http://127.0.0.1:52625/health` and a chat completion.
+6. Verify: `curl -sf http://127.0.0.1:52625/health` and a chat completion.
 
 ## Fallback: vLLM source build for gfx1151
 

scripts/frank-default-large.env

@@ -15,6 +15,10 @@ MODEL_DIR=/home/dank/models/hauhaucs-qwen38
 MODEL_FILE=Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-IQ4_XS.gguf
 DRAFT_FILE=Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-FastMTP-32K.gguf
 # Patched llama.cpp binary. The FastMTP sidecar needs the patch; see AGENTS.md.
+# REQUIRED build flags: -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON.
+# Without FA_ALL_QUANTS the q8_0 KV + flash attention fall back to a slow
+# path and prefill collapses ~12x (95 t/s vs 1300+ t/s at 8K prompt,
+# measured 2026-08-17). Do not rebuild without it.
 LLAMA_BIN=/home/dank/llama.cpp-fastmtp/build/bin/llama-server
 
 ALIAS=frank-default-large

scripts/frank-default-small.env

@@ -38,3 +38,11 @@ GPU_MEMORY_UTILIZATION=0.92
 NGRAM_LOOKUP_MAX=5
 NGRAM_LOOKUP_MIN=1
 NUM_SPEC_TOKENS=5
+
+# RDNA flash-attention for the vision encoder (and general FA paths).
+# Without this, the mm encoder falls back to torch SDPA and the warmup
+# OOMs (256 GiB score matrix at max_model_len; measured 2026-08-17).
+# Requires the flash_attn package with flash_attn_triton_amd findable
+# (the pure-Python copy from the CIRU package plus a flash_attn_2_cuda
+# stub; see docs/frank-redo-2026-08-17.md).
+FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE