dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity
21 Commits 59 Branches 8 Tags
9227e1a8b943cc5ff7a90ee2d1654a9d95cca71c
Commit Graph
7 Commits
Author SHA1 Message Date
Frank a5f72627bb Use --alias frank-default-large, multi-stream validation script 
- Replace --override-kv with --alias for model naming
- Validation script tests 1, 2, 3 parallel streams sequentially
- Model name confirmed: frank-default-large
 2026-08-07 17:15:09 -07:00
Frank 471e963350 Fix parallel validation, remove ngram-mod 
- Rewrite validation script with temp files for true parallel curl requests
- Both prompts now complete 1024 tokens each (was cutting first short)
- Remove ngram-mod from spec pipeline (was -1.2 tok/s regression)
- True parallel throughput: 15.68 tok/s across 2 streams
 2026-08-07 16:44:12 -07:00
Frank 764a65080f Validation: 2 parallel prompts, ~18 tok/s with DFlash 
- Add second parallel prompt (Byzantine Empire history vs ML)
- Both hit 1024 tokens, combined 2048 in 111.6s = 18.34 tok/s
- Parallel throughput matches single-stream baseline
 2026-08-07 16:21:20 -07:00
Frank 25d57184a6 Enable DFlash speculative decoding, remove mlock 
- Add --spec-type draft-dflash (was missing, DFlash was inactive)
- Change --load-mode from mlock to none (RLIMIT_MEMLOCK too small for 2.7GB)
- Increase validation max_tokens from 256 to 1024 with complex ML prompt
- Baseline: 18.76 tok/s (was 7.62 tok/s without DFlash)
 2026-08-07 16:10:55 -07:00
Frank 1949f0651b Pivot to llama.cpp with DFlash speculative decoding on ROCm 
- Abandon vLLM/FP8 (Triton compilation hangs, ~0.5 tok/s on Strix Halo)
- llama.cpp v10133 with ROCm0 offload, DFlash draft model
- Main: unsloth/Qwen3.6-27B-MTP-GGUF:Q8_0 (~22GB)
- Draft: Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp:Q8_0 (~1.8GB)
- 3 parallel streams, 131072 context/stream
- systemd service with minimal sandboxing (ROCm requires /dev/kfd access)
- Baseline: 7.62 tok/s (threshold 5.0 tok/s)
 2026-08-07 15:41:27 -07:00
Frank 2bb1b2ae1d Baseline: ThinkingCap 27B FP8 on dankclaw with enforce-eager and validation timeout 2026-08-07 08:58:20 -07:00
Frank 7dce375ab4 docs: fix dankclaw specs (30GiB RAM, 96GiB GPU-visible, ROCm 7.2.4) + add validate.sh 2026-08-06 22:40:44 -07:00