Frank 1949f0651b Pivot to llama.cpp with DFlash speculative decoding on ROCm ...

- Abandon vLLM/FP8 (Triton compilation hangs, ~0.5 tok/s on Strix Halo)
- llama.cpp v10133 with ROCm0 offload, DFlash draft model
- Main: unsloth/Qwen3.6-27B-MTP-GGUF:Q8_0 (~22GB)
- Draft: Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp:Q8_0 (~1.8GB)
- 3 parallel streams, 131072 context/stream
- systemd service with minimal sandboxing (ROCm requires /dev/kfd access)
- Baseline: 7.62 tok/s (threshold 5.0 tok/s)

2026-08-07 15:41:27 -07:00

..

llama-server.service

Pivot to llama.cpp with DFlash speculative decoding on ROCm

2026-08-07 15:41:27 -07:00

serve-llamacpp-dflash.sh

Pivot to llama.cpp with DFlash speculative decoding on ROCm

2026-08-07 15:41:27 -07:00

validate.sh

Pivot to llama.cpp with DFlash speculative decoding on ROCm

2026-08-07 15:41:27 -07:00