85 lines
3.3 KiB
Markdown
85 lines
3.3 KiB
Markdown
# frank-default-large — Backend and Spec Benchmark
|
|
|
|
This page records the benchmarks that chose the runtime for
|
|
`frank-default-large` on Dankclaw. Every number came from the live
|
|
llama.cpp build on that machine (upstream, b10133+).
|
|
|
|
## Summary
|
|
|
|
- **Backend**: ROCm (HIP). Vulkan is not viable for the full pipeline.
|
|
- **Spec**: DFlash draft model only (`draft-dflash`).
|
|
- **Context**: 3 parallel slots x 200700 (pool 602100).
|
|
- **KV**: `q8_0` K / `f16` V on GPU.
|
|
- **Batch**: 2048 / ubatch 512 (memory-bound, see below).
|
|
|
|
## Backend Comparison (llama-bench, raw target model, no draft)
|
|
|
|
Both backends ran the same target model at 64k context in `llama-bench`
|
|
(`-p 65536 -n 256`).
|
|
|
|
| Device | Prefill 64k (t/s) | Generation 64k (t/s) |
|
|
| -------- | ----------------- | -------------------- |
|
|
| ROCm0 | 220.28 | 6.99 |
|
|
| Vulkan0 | 72.34 | 7.00 |
|
|
|
|
ROCm prefills 3x faster. Generation is a tie in the raw benchmark.
|
|
|
|
## Full-Pipeline Comparison (4k context, real served chain)
|
|
|
|
These used `bench-pipeline-64k.sh` driving `llama-server` through the API at
|
|
4k context, reading the server's own `print_timing` lines.
|
|
|
|
| Spec | Backend | Prefill (t/s) | Generation (t/s) | Draft acceptance |
|
|
| --------------------------- | ------- | ------------- | ---------------- | ---------------- |
|
|
| `draft-dflash` | ROCm | 148.2 | **20.17** | 0.50 |
|
|
| `draft-dflash,ngram-mod` | ROCm | 149.9 | 16.93 | 0.42 |
|
|
| `ngram-mod` | ROCm | 154.0 | 8.68 | 0.57 |
|
|
| `ngram-mod` | Vulkan | 259.6 | 6.90 | 0.17 |
|
|
|
|
### Interpretation
|
|
|
|
- DFlash alone gives the fastest generation on ROCm (20.17 t/s).
|
|
- Layering ngram-mod lowered ROCm generation (16.93 t/s), so it is not
|
|
layered in production. llama.cpp *can* layer DFlash + ngram
|
|
(`--spec-type draft-dflash,ngram-mod`), but it was slower here.
|
|
- Vulkan only works without the DFlash draft. The DFlash `ctx_other`
|
|
setup aborts the Vulkan backend (`ggml-backend.cpp:898: pre-allocated
|
|
tensor ... cannot run the operation (NONE)`). Vulkan ngram generation is
|
|
also slowest (6.90 t/s) with poor acceptance (0.17).
|
|
|
|
**Verdict: ROCm + `draft-dflash`.**
|
|
|
|
## Memory Constraint at 602100 Context
|
|
|
|
At `3 x 200700` context the KV cache alone is ~50 GB and the model is
|
|
29.77 GB, leaving only ~16 GB of the ~96 GB ROCm allocation for compute
|
|
graphs. The compute graph scales with **ubatch**, so:
|
|
|
|
- batch 12288 / ubatch 4096 -> OOM (80.7 GB prefill buffer)
|
|
- batch 6144 / ubatch 2048 -> OOM (40.9 GB prefill buffer)
|
|
- batch 2048 / ubatch 2048 -> OOM (ubatch 2048 too large)
|
|
- batch 2048 / ubatch 512 -> **fits and loads**
|
|
|
|
The requested batch 12288 / ubatch 4096 is not feasible at this context
|
|
size on this hardware.
|
|
|
|
## Command to Reproduce
|
|
|
|
Raw backend bench (ROCm shown; swap `-dev` for the other):
|
|
|
|
```
|
|
llama-bench -m <main>.gguf -p 65536 -n 256 -c 65536 -dev ROCm0 -r 1
|
|
```
|
|
|
|
Full-pipeline bench:
|
|
|
|
```
|
|
./scripts/bench-pipeline-64k.sh ROCm0 draft-dflash
|
|
```
|
|
|
|
## See Also
|
|
|
|
- [[deployment]] — deploy and update process
|
|
- [[DFlash DSpark Speculative Decoding]] — DFlash / DSpark / layering details
|
|
- [[index]] — knowledge base index
|