# frank-default-large — Backend and Spec Benchmark

This page records the benchmarks that chose the runtime for
`frank-default-large` on Dankclaw. Every number came from the live
llama.cpp build on that machine (upstream, b10133+).

## Summary

- **Backend**: ROCm (HIP). Vulkan is not viable for the full pipeline.
- **Spec (2026-08-14)**: MTP sidecar drafter (`draft-mtp`, `a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`, `--spec-draft-n-max 3`). Replaced DFlash, which was retired because it releases a generating slot prematurely under parallel load (see below).
- **Context**: 3 parallel slots x 200700 (pool 602100).
- **KV**: `q8_0` K / `q8_0` V on GPU.
- **Batch**: 8192 / ubatch 8192 (2026-08-14, uses the VRAM headroom; peaks ~65.5 GiB of 96 GiB).

## 2026-08-14: Spec A/B (ROCm, production config, default sampling, 1.1k-token prompt)

| Spec | Generation (t/s) | Parallel-safe |
| ---- | ---------------- | -------------- |
| DFlash (draft-dflash, n_max 8) | ~20.2 (from earlier 4k bench) | **NO** - premature EOS |
| MTP (draft-mtp, n_max 3) | **15.3 - 19.5** | YES |
| MTP + ngram-mod (n_max 3) | 12.0 - 13.6 | YES |
| MTP (draft-mtp, n_max 2) | 12.4 - 14.3 | YES |
| ngram-mod | 6.7 - 8.7 | YES |

Verdict: `draft-mtp` at `n_max 3` is the winner. It nearly matches DFlash's speed, beats ngram-mod by ~2x, and does not exhibit the DFlash parallel bug. Layering ngram under MTP (12-13.6 t/s) does not pay off, the same result as layering ngram under DFlash.

## Incident: DFlash Premature EOS Under Parallel Load (2026-08-14)

Reproduced three times by the user. When a new task lands on a slot with high prompt-cache reuse (LCP similarity ~0.99, `f_keep` ~0.98) while another slot is generating, the generating slot is released ~1 second later with `finish_reason "stop"` and a truncated response. The client records it as a normal completion (no abort). Server log shows a `slot release` with `truncated = 0` and no `cancel task` line.

Mechanism: DFlash's draft model uses one shared draft context (`ctx_dft`) across all parallel slots. When the new task's first DFlash draft pass runs after a large prompt-reuse operation, it corrupts the generating slot's draft state, and the target spuriously samples the EOG token. Known class of bug upstream (`server : do not default to multiple slots with speculative decoding`, PR #17017; DFlash multi-slot routing fixes in the beellama.cpp fork).

Fix: switch away from DFlash. ngram-mod (draftless) and MTP (separate draft machinery) both pass the strict parallel test. MTP was chosen for speed.

## Backend Comparison (llama-bench, raw target model, no draft)

Both backends ran the same target model at 64k context in `llama-bench`
(`-p 65536 -n 256`).

| Device   | Prefill 64k (t/s) | Generation 64k (t/s) |
| -------- | ----------------- | -------------------- |
| ROCm0    | 220.28            | 6.99                 |
| Vulkan0  | 72.34             | 7.00                 |

ROCm prefills 3x faster. Generation is a tie in the raw benchmark.

## Full-Pipeline Comparison (4k context, real served chain)

These used `bench-pipeline-64k.sh` driving `llama-server` through the API at
4k context, reading the server's own `print_timing` lines.

| Spec                        | Backend | Prefill (t/s) | Generation (t/s) | Draft acceptance |
| --------------------------- | ------- | ------------- | ---------------- | ---------------- |
| `draft-dflash`              | ROCm    | 148.2         | **20.17**        | 0.50             |
| `draft-dflash,ngram-mod`    | ROCm    | 149.9         | 16.93            | 0.42             |
| `ngram-mod`                 | ROCm    | 154.0         | 8.68             | 0.57             |
| `ngram-mod`                 | Vulkan  | 259.6         | 6.90             | 0.17             |

### Interpretation

- DFlash alone gives the fastest generation on ROCm (20.17 t/s).
- Layering ngram-mod lowered ROCm generation (16.93 t/s), so it is not
  layered in production. llama.cpp *can* layer DFlash + ngram
  (`--spec-type draft-dflash,ngram-mod`), but it was slower here.
- Vulkan only works without the DFlash draft. The DFlash `ctx_other`
  setup aborts the Vulkan backend (`ggml-backend.cpp:898: pre-allocated
  tensor ... cannot run the operation (NONE)`). Vulkan ngram generation is
  also slowest (6.90 t/s) with poor acceptance (0.17).

**Verdict: ROCm + `draft-dflash`.**

## Memory Constraint at 602100 Context

At `3 x 200700` context the KV cache alone is ~50 GB and the model is
29.77 GB, leaving only ~16 GB of the ~96 GB ROCm allocation for compute
graphs. The compute graph scales with **ubatch**, so:

- batch 12288 / ubatch 4096 -> OOM (80.7 GB prefill buffer)
- batch 6144 / ubatch 2048  -> OOM (40.9 GB prefill buffer)
- batch 2048 / ubatch 2048  -> OOM (ubatch 2048 too large)
- batch 2048 / ubatch 512   -> **fits and loads**

The requested batch 12288 / ubatch 4096 is not feasible at this context
size on this hardware.

## Command to Reproduce

Raw backend bench (ROCm shown; swap `-dev` for the other):

```
llama-bench -m <main>.gguf -p 65536 -n 256 -c 65536 -dev ROCm0 -r 1
```

Full-pipeline bench:

```
./scripts/bench-pipeline-64k.sh ROCm0 draft-dflash
```

## See Also

- [[deployment]] — deploy and update process
- [[DFlash DSpark Speculative Decoding]] — DFlash / DSpark / layering details
- [[index]] — knowledge base index