Frank
25d57184a6
Enable DFlash speculative decoding, remove mlock
...
- Add --spec-type draft-dflash (was missing, DFlash was inactive)
- Change --load-mode from mlock to none (RLIMIT_MEMLOCK too small for 2.7GB)
- Increase validation max_tokens from 256 to 1024 with complex ML prompt
- Baseline: 18.76 tok/s (was 7.62 tok/s without DFlash)
2026-08-07 16:10:55 -07:00
Frank
1949f0651b
Pivot to llama.cpp with DFlash speculative decoding on ROCm
...
- Abandon vLLM/FP8 (Triton compilation hangs, ~0.5 tok/s on Strix Halo)
- llama.cpp v10133 with ROCm0 offload, DFlash draft model
- Main: unsloth/Qwen3.6-27B-MTP-GGUF:Q8_0 (~22GB)
- Draft: Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp:Q8_0 (~1.8GB)
- 3 parallel streams, 131072 context/stream
- systemd service with minimal sandboxing (ROCm requires /dev/kfd access)
- Baseline: 7.62 tok/s (threshold 5.0 tok/s)
2026-08-07 15:41:27 -07:00
Frank
2bb1b2ae1d
Baseline: ThinkingCap 27B FP8 on dankclaw with enforce-eager and validation timeout
2026-08-07 08:58:20 -07:00
Frank
e4fba35e89
Baseline: ThinkingCap 27B FP8 on dankclaw
...
- vLLM 0.26.0+rocm723
- --kv-cache-dtype fp8 --gpu-memory-utilization 0.80
- --max-model-len 32768 --max-num-batched-tokens 2048
- --max-num-seqs 32 --reasoning-parser qwen3 --language-model-only
- Baseline response time: 219s for 50 tokens
2026-08-07 08:49:59 -07:00
Frank
7dce375ab4
docs: fix dankclaw specs (30GiB RAM, 96GiB GPU-visible, ROCm 7.2.4) + add validate.sh
2026-08-06 22:40:44 -07:00
Frank