 Frank
|
ced4c6f46e
|
sync: update vllm-awq-qwen36-dflash-ampere.sh from dankserver
- Model: Avesed/Qwen3.6-35B-A3B-INT8-W8A8 (was AWQ 4bit)
- Image: vllm/vllm-openai:nightly (was :latest)
- Spec method: mtp with 2 tokens (was dflash with 12)
- Max model len: 200700 (was 32768)
- Max num seqs: 3 (was 1)
- Max batched tokens: 3072 (was 4096)
- GPU mem util: 0.97 (was 0.96)
- CPU offload: 30GB (was 40GB)
- Add --enable-prefix-caching
- Add --kv-cache-dtype auto
|
2026-08-10 03:23:09 -07:00 |
|
|
Frank
|
11d66c3823
|
Set max-model-len to 32768 for testing
|
2026-08-09 22:19:32 -07:00 |
|
|
Frank
|
6bdabfc65b
|
Add VLLM_USE_V2_MODEL_RUNNER=0 to skip V2 model runner MARLIN repacking OOM
|
2026-08-09 21:49:25 -07:00 |
|
|
Frank
|
83b0bea29b
|
Fix: move commented speculative config to end to not break bash line continuation
|
2026-08-09 12:27:28 -07:00 |
|
|
Frank
|
88aa18ef34
|
Reduce GPU memory to 75%, add expandable_segments to fix OOM
|
2026-08-09 12:11:48 -07:00 |
|
|
Frank
|
9dfa16fae9
|
Disable speculative decoding - OOM on stable vLLM too
|
2026-08-09 12:08:26 -07:00 |
|
|
Frank
|
578c3e8f71
|
Enable DFlash speculative decoding (12 tokens), increase CPU offload to 20GB
|
2026-08-09 11:54:40 -07:00 |
|
|
Frank
|
8c13c7bbaa
|
Add AGENTS.md git workflow rules and initial vllm-awq-qwen36-dflash-ampere.sh from dankserver
|
2026-08-09 11:54:04 -07:00 |
|