dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Update documentation with AWQ+DFlash lessons learned and improved validation script

Browse Source
This commit is contained in:
Frank committed 2026-08-08 21:29:23 -07:00
1 parent c530e16880
commit 5ed9bf0410
3 files changed
+115 -13
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+35 -1
View File
@@ -15,12 +15,46 @@ Scripts for running large language models via vLLM on two local machines.
|-------|--------|--------|
| Qwen3.6 35B A3B FP8 | dankserver | FP8 |
| Qwen3.6 27B FP8 | dankclaw | FP8 |
| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ 4bit |
| Qwen3.6 35B A3B DFlash | dankserver | Draft model |
## Structure
- `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, and deployment notes
- `scripts/` — Executable vLLM launch scripts (coming soon)
- `scripts/` — Executable vLLM launch scripts
- `vllm-qwen36.sh` — FP8 inference script for dankserver
- `vllm-awq-qwen36-dflash.sh` — AWQ 4bit with DFlash speculative decoding (docker)
- `vllm-awq-qwen36-dflash-ampere.sh` — AWQ 4bit with DFlash speculative decoding (Ampere-optimized, systemd)
## Knowledge Base
See [[docs/README.md]] for the project knowledge base index.
## Lessons Learned
### AWQ + DFlash on Ampere Hardware (2026-08-08)
**What worked:**
- `cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit` loads successfully on Ampere GPUs
- AWQ Marlin kernel engaged automatically for weight conversion
- DFlash draft model `z-lab/Qwen3.6-35B-A3B-DFlash` detected as EAGLE model
- torch.compile with cudagraphs works with `--max-num-batched-tokens 1536`
**What didn't work:**
- `--enable-prefix-caching` conflicts with Mamba cache "align" mode — causes `AssertionError` in `unify_kv_cache_spec_page_size`
- `--max-model-len 262144` causes OOM during AWQ→Marlin weight conversion on 24GB GPU
- Nightly vLLM `--speculative-config` requires `num_speculative_tokens` when model is provided
- `--speculative-model` flag removed in nightly; use `--speculative-config` JSON instead
**Working configuration:**
- `--max-model-len 32768` (32k context)
- `--cpu-offload-gb 40` with `--cpu-offload-params "experts"`
- `--gpu-memory-utilization 0.95`
- `--performance-mode throughput`
- `--enable-chunked-prefill`
- `--max-num-batched-tokens 1536`
- Thread binding: `OMP_NUM_THREADS=8`, `VLLM_CPU_OMP_THREADS_BIND=8`
**Still needs work:**
- DFlash speculative decoding integration (KV cache assertion bug in nightly)
- Full context length support (32k is temporary workaround for OOM)