llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
This commit is contained in:
Frank committed
1 parent
5ced474eb9
commit
55f87fcdce
195 files changed
+2085
-2064
No files matched your search
@@ -0,0 +1,128 @@
|
||||
---
|
||||
type: reference
|
||||
description: "This page records the benchmarks that chose the runtime for `frank-dragoon` on Dankclaw."
|
||||
tags:
|
||||
- llm
|
||||
- frank
|
||||
timestamp: 2026-09-03
|
||||
---
|
||||
|
||||
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
|
||||
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
|
||||
> [deployment.md](deployment.md) for the active configuration.
|
||||
|
||||
# frank-dragoon — Backend and Spec Benchmark
|
||||
|
||||
This page records the benchmarks that chose the runtime for
|
||||
`frank-dragoon` on Dankclaw. Every number came from the live
|
||||
llama.cpp build on that machine (upstream, b10133+).
|
||||
|
||||
## Summary
|
||||
|
||||
- **Backend**: **Vulkan0** (2026-08-15). At 262k ctx Vulkan decodes 48%
|
||||
faster than ROCm (5.68 vs 3.83 t/s); ROCm only wins prefill 119 vs 106
|
||||
t/s. Generation is the interactive cost, so Vulkan0.
|
||||
- **Spec (2026-08-15)**: llama.cpp `--spec-default` preset (ngram-mod,
|
||||
n_match 24 / n_min 48 / n_max 64, draftless). Replaced DSpark, which
|
||||
crashed the ROCm backend under two concurrent 16k prompts (see
|
||||
[[qwen3.8-spec-matrix]]).
|
||||
- **Parallel**: 2 slots — passes the 2x32k parallel stream test.
|
||||
- **Context**: 2 slots x 262144.
|
||||
- **KV**: `q8_0` K / `q8_0` V on GPU.
|
||||
- **Batch**: 8192 / ubatch 256.
|
||||
|
||||
> The A/B tables below are retained for history but predate the 2026-08-14
|
||||
> Qwen3.8-27B `UD-Q8_K_XL` switch. The current production spec decision,
|
||||
> the DSpark parallel crash, and the high-context ROCm-vs-Vulkan bench are
|
||||
> recorded on [[qwen3.8-spec-matrix]].
|
||||
|
||||
## 2026-08-14: Spec A/B (ROCm, production config, default sampling, 1.1k-token prompt)
|
||||
|
||||
| Spec | Generation (t/s) | Parallel-safe |
|
||||
| ---- | ---------------- | -------------- |
|
||||
| DFlash (draft-dflash, n_max 8) | ~20.2 (from earlier 4k bench) | **NO** - premature EOS |
|
||||
| MTP (draft-mtp, n_max 3) | **15.3 - 19.5** | YES |
|
||||
| MTP + ngram-mod (n_max 3) | 12.0 - 13.6 | YES |
|
||||
| MTP (draft-mtp, n_max 2) | 12.4 - 14.3 | YES |
|
||||
| ngram-mod | 6.7 - 8.7 | YES |
|
||||
|
||||
Verdict: `draft-mtp` at `n_max 3` is the winner. It nearly matches DFlash's speed, beats ngram-mod by ~2x, and does not exhibit the DFlash parallel bug. Layering ngram under MTP (12-13.6 t/s) does not pay off, the same result as layering ngram under DFlash.
|
||||
|
||||
## Incident: DFlash Premature EOS Under Parallel Load (2026-08-14)
|
||||
|
||||
Reproduced three times by the user. When a new task lands on a slot with high prompt-cache reuse (LCP similarity ~0.99, `f_keep` ~0.98) while another slot is generating, the generating slot is released ~1 second later with `finish_reason "stop"` and a truncated response. The client records it as a normal completion (no abort). Server log shows a `slot release` with `truncated = 0` and no `cancel task` line.
|
||||
|
||||
Mechanism: DFlash's draft model uses one shared draft context (`ctx_dft`) across all parallel slots. When the new task's first DFlash draft pass runs after a large prompt-reuse operation, it corrupts the generating slot's draft state, and the target spuriously samples the EOG token. Known class of bug upstream (`server : do not default to multiple slots with speculative decoding`, PR #17017; DFlash multi-slot routing fixes in the beellama.cpp fork).
|
||||
|
||||
Fix: switch away from DFlash. ngram-mod (draftless) and MTP (separate draft machinery) both pass the strict parallel test. MTP was chosen for speed.
|
||||
|
||||
## Backend Comparison (llama-bench, raw target model, no draft)
|
||||
|
||||
Both backends ran the same target model at 64k context in `llama-bench`
|
||||
(`-p 65536 -n 256`).
|
||||
|
||||
| Device | Prefill 64k (t/s) | Generation 64k (t/s) |
|
||||
| -------- | ----------------- | -------------------- |
|
||||
| ROCm0 | 220.28 | 6.99 |
|
||||
| Vulkan0 | 72.34 | 7.00 |
|
||||
|
||||
ROCm prefills 3x faster. Generation is a tie in the raw benchmark.
|
||||
|
||||
## Full-Pipeline Comparison (4k context, real served chain)
|
||||
|
||||
These used `bench-pipeline-64k.sh` driving `llama-server` through the API at
|
||||
4k context, reading the server's own `print_timing` lines.
|
||||
|
||||
| Spec | Backend | Prefill (t/s) | Generation (t/s) | Draft acceptance |
|
||||
| --------------------------- | ------- | ------------- | ---------------- | ---------------- |
|
||||
| `draft-dflash` | ROCm | 148.2 | **20.17** | 0.50 |
|
||||
| `draft-dflash,ngram-mod` | ROCm | 149.9 | 16.93 | 0.42 |
|
||||
| `ngram-mod` | ROCm | 154.0 | 8.68 | 0.57 |
|
||||
| `ngram-mod` | Vulkan | 259.6 | 6.90 | 0.17 |
|
||||
|
||||
### Interpretation
|
||||
|
||||
- DFlash alone gives the fastest generation on ROCm (20.17 t/s).
|
||||
- Layering ngram-mod lowered ROCm generation (16.93 t/s), so it is not
|
||||
layered in production. llama.cpp *can* layer DFlash + ngram
|
||||
(`--spec-type draft-dflash,ngram-mod`), but it was slower here.
|
||||
- Vulkan only works without the DFlash draft. The DFlash `ctx_other`
|
||||
setup aborts the Vulkan backend (`ggml-backend.cpp:898: pre-allocated
|
||||
tensor ... cannot run the operation (NONE)`). Vulkan ngram generation is
|
||||
also slowest (6.90 t/s) with poor acceptance (0.17).
|
||||
|
||||
**Verdict: ROCm + `draft-dflash`.**
|
||||
|
||||
## Memory Constraint at 602100 Context
|
||||
|
||||
At `3 x 200700` context the KV cache alone is ~50 GB and the model is
|
||||
29.77 GB, leaving only ~16 GB of the ~96 GB ROCm allocation for compute
|
||||
graphs. The compute graph scales with **ubatch**, so:
|
||||
|
||||
- batch 12288 / ubatch 4096 -> OOM (80.7 GB prefill buffer)
|
||||
- batch 6144 / ubatch 2048 -> OOM (40.9 GB prefill buffer)
|
||||
- batch 2048 / ubatch 2048 -> OOM (ubatch 2048 too large)
|
||||
- batch 2048 / ubatch 512 -> **fits and loads**
|
||||
|
||||
The requested batch 12288 / ubatch 4096 is not feasible at this context
|
||||
size on this hardware.
|
||||
|
||||
## Command to Reproduce
|
||||
|
||||
Raw backend bench (ROCm shown; swap `-dev` for the other):
|
||||
|
||||
```
|
||||
llama-bench -m <main>.gguf -p 65536 -n 256 -c 65536 -dev ROCm0 -r 1
|
||||
```
|
||||
|
||||
Full-pipeline bench:
|
||||
|
||||
```
|
||||
./scripts/bench-pipeline-64k.sh ROCm0 draft-dflash
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [[deployment]] — deploy and update process
|
||||
- [[DFlash DSpark Speculative Decoding]] — DFlash / DSpark / layering details
|
||||
- [[index]] — knowledge base index
|
||||
Reference in new issue
Block a user