Update README with validation instructions and performance data
This commit is contained in:
1 parent
726b9a35ec
commit
469bfaf979
2 files changed
+93
-31
No files matched your search
@@ -55,6 +55,34 @@ See [[docs/README.md]] for the project knowledge base index.
|
||||
- `--max-num-batched-tokens 1536`
|
||||
- Thread binding: `OMP_NUM_THREADS=8`, `VLLM_CPU_OMP_THREADS_BIND=8`
|
||||
|
||||
**Performance observations (no speculative decoding):**
|
||||
| Streams | Prompt Speed | Completion Speed | Total Time |
|
||||
|---------|-------------|-----------------|------------|
|
||||
| 1 | 4.27 tok/s | 21.42 tok/s | 11,951ms |
|
||||
| 2 | 2.44 tok/s | 12.25 tok/s | 20,891ms |
|
||||
| 3 | 1.35 tok/s | 6.76 tok/s | 37,849ms |
|
||||
| 4 | 1.43 tok/s | 7.19 tok/s | 35,606ms |
|
||||
|
||||
Note: Speeds decrease with more streams due to KV cache contention.
|
||||
|
||||
**Still needs work:**
|
||||
- DFlash speculative decoding integration (KV cache assertion bug in nightly)
|
||||
- Full context length support (32k is temporary workaround for OOM)
|
||||
|
||||
## Validation
|
||||
|
||||
Use `scripts/validate-vllm.sh` to test inference servers:
|
||||
|
||||
```bash
|
||||
# Test dankserver (default)
|
||||
bash scripts/validate-vllm.sh -t dankserver
|
||||
|
||||
# Test dankclaw
|
||||
bash scripts/validate-vllm.sh -t dankclaw
|
||||
|
||||
# Custom host/port
|
||||
bash scripts/validate-vllm.sh -H 192.168.1.100 -P 8001 -M my-model
|
||||
|
||||
# Run tests
|
||||
bash tests/test_validator.sh dankserver
|
||||
```
|
||||
Reference in new issue
Block a user