Frank
841d872aa7
sync scripts from remotes: ROCmFPX llama-server.service (dankclaw), dankserver vLLM serve/validate; drop failed-attempt scripts
...
- llama-server.service: dankclaw migrated to ROCmFPX runtime (Ling-3.0-flash,
draft-mtp, no-mmap/direct-io) from regular llama.cpp Qwen3.6-27B-DFlash
- serve-vllm-qwen36-dflash.sh: dankserver version (relies on image 'vllm serve'
entrypoint; HF cache mounted read-write)
- validate-vllm.sh: dankserver multi-stream vLLM validation (4 unique prompts,
1-4 streams, prompt+completion speed)
- remove superseded scripts: serve-llamacpp-dflash.sh, validate.sh,
vllm-awq-qwen36-dflash-ampere.sh, vllm-qwen36.service
2026-08-10 19:48:46 -07:00
Frank
663e47866b
Bump context to 600000 (200000/stream x 3 parallel)
2026-08-07 17:18:10 -07:00
Frank
a5f72627bb
Use --alias frank-default-large, multi-stream validation script
...
- Replace --override-kv with --alias for model naming
- Validation script tests 1, 2, 3 parallel streams sequentially
- Model name confirmed: frank-default-large
2026-08-07 17:15:09 -07:00
Frank
471e963350
Fix parallel validation, remove ngram-mod
...
- Rewrite validation script with temp files for true parallel curl requests
- Both prompts now complete 1024 tokens each (was cutting first short)
- Remove ngram-mod from spec pipeline (was -1.2 tok/s regression)
- True parallel throughput: 15.68 tok/s across 2 streams
2026-08-07 16:44:12 -07:00
Frank
2426ca563a
Add ngram-mod to spec pipeline: -1.2 tok/s regression
...
- DFlash+ngram-mod: 17.11 tok/s vs DFlash alone: 18.34 tok/s
- ngram overhead not offset by acceptance rate at current settings
- Keeping in place for now, may tune ngram-mod params later
2026-08-07 16:25:49 -07:00
Frank
25d57184a6
Enable DFlash speculative decoding, remove mlock
...
- Add --spec-type draft-dflash (was missing, DFlash was inactive)
- Change --load-mode from mlock to none (RLIMIT_MEMLOCK too small for 2.7GB)
- Increase validation max_tokens from 256 to 1024 with complex ML prompt
- Baseline: 18.76 tok/s (was 7.62 tok/s without DFlash)
2026-08-07 16:10:55 -07:00
Frank
1949f0651b
Pivot to llama.cpp with DFlash speculative decoding on ROCm
...
- Abandon vLLM/FP8 (Triton compilation hangs, ~0.5 tok/s on Strix Halo)
- llama.cpp v10133 with ROCm0 offload, DFlash draft model
- Main: unsloth/Qwen3.6-27B-MTP-GGUF:Q8_0 (~22GB)
- Draft: Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp:Q8_0 (~1.8GB)
- 3 parallel streams, 131072 context/stream
- systemd service with minimal sandboxing (ROCm requires /dev/kfd access)
- Baseline: 7.62 tok/s (threshold 5.0 tok/s)
2026-08-07 15:41:27 -07:00
Frank