Files

T

Frank d9af7f1d19 fleet redo 2026-08-17: retire PP+NPU, new topology

- Move all previous scripts to old/scripts/ (retired PP/NPU-era)
- frank-default-large: llama.cpp FastMTP on dankserver (HauhauCS
  Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod, K q8_0/V f16,
  128K ctx, batch 2048/ubatch 1024, port 7770)
- frank-default-small: vLLM ROCm on dankclaw (Avesed Qwen3.6-35B-A3B
  INT8 W8A8, full 262144 ctx, no KV quant, 4 seqs, batch 16384,
  ngram spec, port 52625)
- retarget deploy.sh (dankserver/dankclaw, inline sudo, no helpers)
- docs: deployment + index + AGENTS.md + README for new topology,
  superseded banners on retired docs, add frank-redo-2026-08-17 runbook

2026-08-17 15:23:46 -07:00

5.8 KiB

Raw Blame History

SUPERSEDED 2026-08-17. The Frank redo retired this topology. See frank-redo-2026-08-17.md and deployment.md for the active configuration.

frank-default-large — Backend and Spec Benchmark

This page records the benchmarks that chose the runtime for frank-default-large on Dankclaw. Every number came from the live llama.cpp build on that machine (upstream, b10133+).

Summary

Backend: Vulkan0 (2026-08-15). At 262k ctx Vulkan decodes 48% faster than ROCm (5.68 vs 3.83 t/s); ROCm only wins prefill 119 vs 106 t/s. Generation is the interactive cost, so Vulkan0.
Spec (2026-08-15): llama.cpp --spec-default preset (ngram-mod, n_match 24 / n_min 48 / n_max 64, draftless). Replaced DSpark, which crashed the ROCm backend under two concurrent 16k prompts (see qwen3.8-spec-matrix).
Parallel: 2 slots — passes the 2x32k parallel stream test.
Context: 2 slots x 262144.
KV: q8_0 K / q8_0 V on GPU.
Batch: 8192 / ubatch 256.

The A/B tables below are retained for history but predate the 2026-08-14 Qwen3.8-27B UD-Q8_K_XL switch. The current production spec decision, the DSpark parallel crash, and the high-context ROCm-vs-Vulkan bench are recorded on qwen3.8-spec-matrix.

2026-08-14: Spec A/B (ROCm, production config, default sampling, 1.1k-token prompt)

Spec	Generation (t/s)	Parallel-safe
DFlash (draft-dflash, n_max 8)	~20.2 (from earlier 4k bench)	NO - premature EOS
MTP (draft-mtp, n_max 3)	15.3 - 19.5	YES
MTP + ngram-mod (n_max 3)	12.0 - 13.6	YES
MTP (draft-mtp, n_max 2)	12.4 - 14.3	YES
ngram-mod	6.7 - 8.7	YES

Verdict: draft-mtp at n_max 3 is the winner. It nearly matches DFlash's speed, beats ngram-mod by ~2x, and does not exhibit the DFlash parallel bug. Layering ngram under MTP (12-13.6 t/s) does not pay off, the same result as layering ngram under DFlash.

Incident: DFlash Premature EOS Under Parallel Load (2026-08-14)

Reproduced three times by the user. When a new task lands on a slot with high prompt-cache reuse (LCP similarity ~0.99, f_keep ~0.98) while another slot is generating, the generating slot is released ~1 second later with finish_reason "stop" and a truncated response. The client records it as a normal completion (no abort). Server log shows a slot release with truncated = 0 and no cancel task line.

Mechanism: DFlash's draft model uses one shared draft context (ctx_dft) across all parallel slots. When the new task's first DFlash draft pass runs after a large prompt-reuse operation, it corrupts the generating slot's draft state, and the target spuriously samples the EOG token. Known class of bug upstream (server : do not default to multiple slots with speculative decoding, PR #17017; DFlash multi-slot routing fixes in the beellama.cpp fork).

Fix: switch away from DFlash. ngram-mod (draftless) and MTP (separate draft machinery) both pass the strict parallel test. MTP was chosen for speed.

Backend Comparison (llama-bench, raw target model, no draft)

Both backends ran the same target model at 64k context in llama-bench (-p 65536 -n 256).

Device	Prefill 64k (t/s)	Generation 64k (t/s)
ROCm0	220.28	6.99
Vulkan0	72.34	7.00

ROCm prefills 3x faster. Generation is a tie in the raw benchmark.

Full-Pipeline Comparison (4k context, real served chain)

These used bench-pipeline-64k.sh driving llama-server through the API at 4k context, reading the server's own print_timing lines.

Spec	Backend	Prefill (t/s)	Generation (t/s)	Draft acceptance
`draft-dflash`	ROCm	148.2	20.17	0.50
`draft-dflash,ngram-mod`	ROCm	149.9	16.93	0.42
`ngram-mod`	ROCm	154.0	8.68	0.57
`ngram-mod`	Vulkan	259.6	6.90	0.17

Interpretation

DFlash alone gives the fastest generation on ROCm (20.17 t/s).
Layering ngram-mod lowered ROCm generation (16.93 t/s), so it is not layered in production. llama.cpp can layer DFlash + ngram (--spec-type draft-dflash,ngram-mod), but it was slower here.
Vulkan only works without the DFlash draft. The DFlash ctx_other setup aborts the Vulkan backend (ggml-backend.cpp:898: pre-allocated tensor ... cannot run the operation (NONE)). Vulkan ngram generation is also slowest (6.90 t/s) with poor acceptance (0.17).

Verdict: ROCm + draft-dflash.

Memory Constraint at 602100 Context

At 3 x 200700 context the KV cache alone is ~50 GB and the model is 29.77 GB, leaving only ~16 GB of the ~96 GB ROCm allocation for compute graphs. The compute graph scales with ubatch, so:

batch 12288 / ubatch 4096 -> OOM (80.7 GB prefill buffer)
batch 6144 / ubatch 2048 -> OOM (40.9 GB prefill buffer)
batch 2048 / ubatch 2048 -> OOM (ubatch 2048 too large)
batch 2048 / ubatch 512 -> fits and loads

The requested batch 12288 / ubatch 4096 is not feasible at this context size on this hardware.

Command to Reproduce

Raw backend bench (ROCm shown; swap -dev for the other):

llama-bench -m <main>.gguf -p 65536 -n 256 -c 65536 -dev ROCm0 -r 1

Full-pipeline bench:

./scripts/bench-pipeline-64k.sh ROCm0 draft-dflash

5.8 KiB Raw Blame History