frank-default-large — Backend and Spec Benchmark

This page records the benchmarks that chose the runtime for frank-default-large on Dankclaw. Every number came from the live llama.cpp build on that machine (upstream, b10133+).

Summary

Backend: ROCm (HIP). Vulkan is not viable for the full pipeline.
Spec: DFlash draft model only (draft-dflash).
Context: 3 parallel slots x 200700 (pool 602100).
KV: q8_0 K / f16 V on GPU.
Batch: 2048 / ubatch 512 (memory-bound, see below).

Backend Comparison (llama-bench, raw target model, no draft)

Both backends ran the same target model at 64k context in llama-bench (-p 65536 -n 256).

Device	Prefill 64k (t/s)	Generation 64k (t/s)
ROCm0	220.28	6.99
Vulkan0	72.34	7.00

ROCm prefills 3x faster. Generation is a tie in the raw benchmark.

Full-Pipeline Comparison (4k context, real served chain)

These used bench-pipeline-64k.sh driving llama-server through the API at 4k context, reading the server's own print_timing lines.

Spec	Backend	Prefill (t/s)	Generation (t/s)	Draft acceptance
`draft-dflash`	ROCm	148.2	20.17	0.50
`draft-dflash,ngram-mod`	ROCm	149.9	16.93	0.42
`ngram-mod`	ROCm	154.0	8.68	0.57
`ngram-mod`	Vulkan	259.6	6.90	0.17

Interpretation

DFlash alone gives the fastest generation on ROCm (20.17 t/s).
Layering ngram-mod lowered ROCm generation (16.93 t/s), so it is not layered in production. llama.cpp can layer DFlash + ngram (--spec-type draft-dflash,ngram-mod), but it was slower here.
Vulkan only works without the DFlash draft. The DFlash ctx_other setup aborts the Vulkan backend (ggml-backend.cpp:898: pre-allocated tensor ... cannot run the operation (NONE)). Vulkan ngram generation is also slowest (6.90 t/s) with poor acceptance (0.17).

Verdict: ROCm + draft-dflash.

Memory Constraint at 602100 Context

At 3 x 200700 context the KV cache alone is ~50 GB and the model is 29.77 GB, leaving only ~16 GB of the ~96 GB ROCm allocation for compute graphs. The compute graph scales with ubatch, so:

batch 12288 / ubatch 4096 -> OOM (80.7 GB prefill buffer)
batch 6144 / ubatch 2048 -> OOM (40.9 GB prefill buffer)
batch 2048 / ubatch 2048 -> OOM (ubatch 2048 too large)
batch 2048 / ubatch 512 -> fits and loads

The requested batch 12288 / ubatch 4096 is not feasible at this context size on this hardware.

Command to Reproduce

Raw backend bench (ROCm shown; swap -dev for the other):

llama-bench -m <main>.gguf -p 65536 -n 256 -c 65536 -dev ROCm0 -r 1

Full-pipeline bench:

./scripts/bench-pipeline-64k.sh ROCm0 draft-dflash

3.3 KiB Raw Blame History