Update README with validation instructions and performance data

This commit is contained in:
Frank committed 2026-08-08 21:46:00 -07:00
1 parent 726b9a35ec
commit 469bfaf979
2 files changed
+93 -31

No files matched your search

+28
View File
@@ -55,6 +55,34 @@ See [[docs/README.md]] for the project knowledge base index.
- `--max-num-batched-tokens 1536`
- Thread binding: `OMP_NUM_THREADS=8`, `VLLM_CPU_OMP_THREADS_BIND=8`
**Performance observations (no speculative decoding):**
| Streams | Prompt Speed | Completion Speed | Total Time |
|---------|-------------|-----------------|------------|
| 1 | 4.27 tok/s | 21.42 tok/s | 11,951ms |
| 2 | 2.44 tok/s | 12.25 tok/s | 20,891ms |
| 3 | 1.35 tok/s | 6.76 tok/s | 37,849ms |
| 4 | 1.43 tok/s | 7.19 tok/s | 35,606ms |
Note: Speeds decrease with more streams due to KV cache contention.
**Still needs work:**
- DFlash speculative decoding integration (KV cache assertion bug in nightly)
- Full context length support (32k is temporary workaround for OOM)
## Validation
Use `scripts/validate-vllm.sh` to test inference servers:
```bash
# Test dankserver (default)
bash scripts/validate-vllm.sh -t dankserver
# Test dankclaw
bash scripts/validate-vllm.sh -t dankclaw
# Custom host/port
bash scripts/validate-vllm.sh -H 192.168.1.100 -P 8001 -M my-model
# Run tests
bash tests/test_validator.sh dankserver
```