docs: boot policy (only PP units enabled), deploy.sh dankserver hazard, stale-claim cleanup
- deployment.md: Boot Policy (2026-08-17) section (disabled units table, re-enable recipes, stale frank-restart-small.sh helper, broken deploy.sh dankserver target); Update Dankserver -> frank-pp-worker; condensed retired 35B history block and 2026-08-13 incident - AGENTS.md: boot-policy note, target map fix (Q6_K, split 0.68), deploy.sh dankserver warning, obsolete quirk entries condensed - README.md: rewrite active services/models tables (PP head + worker) - index.md: topology fix (qwythos:9b, Q6_K split 0.68, NPU disabled) - qwen3.8-spec-matrix.md: superseded configs merged into compact history; stale inline env block removed; bundled-head MTP note - FastFlowLM, PP Deployment: boot-policy + enablement notes - System Specs x2, DFlash doc, 35B FP8 doc, Ling docs: stale active claims fixed, retired banners, never-shipped plan stubbed
This commit is contained in:
Frank committed
1 parent
3f88b8e1ed
commit
968066ff23
13 files changed
+190
-246
No files matched your search
+43
-117
@@ -1,9 +1,8 @@
|
||||
# Qwen3.8-27B UD-Q8_K_XL Speculative-Decoding Matrix
|
||||
# Qwen3.8-27B Speculative-Decoding Matrix
|
||||
|
||||
This page records the speculative-decoding benchmark and the production
|
||||
tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service
|
||||
(`frank-default-large`), now a pipeline-parallel head with a Dankserver
|
||||
worker.
|
||||
This page records the speculative-decoding benchmark history and the
|
||||
production tuning for the Qwen3.8-27B service (`frank-default-large`),
|
||||
now a pipeline-parallel head with a Dankserver worker.
|
||||
|
||||
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
|
||||
|
||||
@@ -42,99 +41,44 @@ Current production, verified against the live process and journal
|
||||
- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`,
|
||||
`--gpu-layers all`.
|
||||
|
||||
## Result (superseded 2026-08-17: Q8_K_XL, f16/f16 KV, split 0.53)
|
||||
|
||||
Superseded for topology on 2026-08-16, when the PP split became
|
||||
production. The ngram-mod preset survives as the fallback layer under
|
||||
MTP. The Vulkan0 verdict in this section applies to the standalone
|
||||
configuration only.
|
||||
|
||||
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
|
||||
- Spec: llama.cpp **`--spec-default`** preset = **ngram-mod** (n_match 24,
|
||||
n_min 48, n_max 64). No draft model. Chosen 2026-08-15 after DSpark
|
||||
crashed under parallel load (see below).
|
||||
- **PARALLEL=2** — passes the 2x32k parallel stream test (both streams
|
||||
complete 256/256 concurrently, no crash). ngram-mod is draftless and
|
||||
parallel-safe.
|
||||
- **Backend: Vulkan0** — at 262144 ctx (~200k prompt) Vulkan decodes
|
||||
5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm only wins prefill (119 vs
|
||||
106 t/s). See the high-context bench table below.
|
||||
- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).
|
||||
|
||||
## Result (superseded 2026-08-15: DSpark)
|
||||
|
||||
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
|
||||
- Spec: **DSpark sidecar draft** `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
|
||||
(`Qwen3.8-27B-DSpark-Q8_0.gguf`, 1.36B, block_size 7), `--spec-type
|
||||
draft-dspark`, `--spec-draft-n-max 7` (clamps to block_size).
|
||||
- **PARALLEL=1.** Two concurrent 16k prose prompts crashed the ROCm backend
|
||||
with `ROCm error: an illegal memory access` during per-sequence block
|
||||
sampling. See the DSpark parallel incident below.
|
||||
- Retired the same day in favor of `--spec-default`.
|
||||
- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).
|
||||
|
||||
## Result (superseded 2026-08-14: bundled MTP)
|
||||
|
||||
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
|
||||
- Spec: `draft-mtp`, `--spec-draft-n-max 5`, no `--spec-draft-p-min`.
|
||||
- Batch 8192 / ubatch 256, parallel 2, ctx 524288 (262144 per slot).
|
||||
- Realistic-prompt generation: ~13.3 t/s single, ~8 t/s per slot under
|
||||
2 concurrent streams. No premature stop.
|
||||
|
||||
## Incident: DSpark ROCm Crash Under Parallel Load (2026-08-15)
|
||||
|
||||
A two-stream parallel acceptance test crashed `frank-default-large`.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Two concurrent 16k prose prompts<br>high prompt-cache reuse] --> B[ROCm illegal memory access<br>in common_speculative_impl_draft_dflash::draft]
|
||||
B --> C[status 6/ABRT core dump]
|
||||
C --> D[systemd auto-restart, 503 during reload]
|
||||
D --> E[pinned PARALLEL=1]
|
||||
```
|
||||
|
||||
Trigger: two distinct-but-overlapping ~16k-token realistic prose prompts
|
||||
started simultaneously on PARALLEL=2. The ROCm backend aborted at
|
||||
`ggml-cuda.cu:106` with `ROCm error: an illegal memory access was
|
||||
encountered`, surfacing in `hipStreamSynchronize` inside
|
||||
`common_speculative_impl_draft_dflash::draft` (the class used by
|
||||
`draft-dspark`). `systemd` caught it (status 6/ABRT) and restarted the
|
||||
unit; in-flight streams saw `Remote end closed connection without response`
|
||||
until the ~reload finished.
|
||||
|
||||
Mechanism: DSpark reuses the DFlash block-diffusion draft machinery. The
|
||||
DFlash draft model uses one shared draft context (`ctx_dft`) across all
|
||||
parallel slots; under concurrent block drafting on ROCm it performs an
|
||||
illegal memory access (HIP stream sync). This is the same multi-slot
|
||||
class of bug that retired `draft-dflash` for premature EOS; here it is a
|
||||
hard ROCm crash instead of a bad sample.
|
||||
|
||||
Fix: pin **PARALLEL=1** (single slot). With one slot, concurrent requests
|
||||
queue safely and each completes to `max_tokens`. Revisit after an upstream
|
||||
multi-slot DSpark fix (DSpark is known-good single-slot). Single-slot 16k
|
||||
prose generation was ~steady (measuring only the correct-size signal, not
|
||||
this page).
|
||||
|
||||
## Parallel results by stream count (2026-08-15, correct-size tests)
|
||||
|
||||
| Parallel | Two 16k prose prompts | Two 67k prose prompts |
|
||||
| -------- | -------------------- | --------------------- |
|
||||
| 2 | crashes (illegal mem access) | passes |
|
||||
| 1 | queues, both complete | n/a |
|
||||
|
||||
Note: the crash manifests on the smaller (16k) concurrent streams, not the
|
||||
larger 67k pair. Size is not the trigger; concurrent block drafting on the
|
||||
shared draft context is.
|
||||
|
||||
## How MTP works on Qwen3.8
|
||||
## How MTP works on Qwen3.8 (bundled-head era)
|
||||
|
||||
The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside
|
||||
the model file. llama.cpp b10411+ auto-detects it and creates the draft
|
||||
context from the target model. Do not pass `--spec-draft-model`.
|
||||
context from the target model. Do not pass `--spec-draft-model` for those
|
||||
files. This is superseded: the production `Qwen3.8-27B-Uncensored-Q6_K.gguf`
|
||||
has NO bundled head, and the serve script passes the standalone draft
|
||||
sidecar with `--spec-draft-model` (see Production state above).
|
||||
|
||||
The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild
|
||||
on Dankclaw is b10433 (`9b05354ec`) with `GGML_HIP` + `gfx1151` + Vulkan.
|
||||
|
||||
## History (superseded configurations, 2026-08-14 to 08-16)
|
||||
|
||||
All superseded configs used `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB,
|
||||
bundled MTP) before the Q6_K swap. The ngram-mod preset survives as the
|
||||
fallback layer under MTP. The Vulkan0 verdict from this era applies to the
|
||||
standalone configuration only; the PP head runs ROCm0.
|
||||
|
||||
- **2026-08-14: bundled MTP** — `draft-mtp`, n_max 5, parallel 2, ctx
|
||||
524288 (2 x 262144). ~13.3 t/s single, ~8 t/s per slot under 2
|
||||
concurrent streams. No premature stop.
|
||||
- **2026-08-15: DSpark sidecar** — `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
|
||||
(Q8_0, 1.36B, block_size 7), n_max 7, PARALLEL=1. Two concurrent 16k
|
||||
prose prompts crashed the ROCm backend with `ROCm error: an illegal
|
||||
memory access` in `common_speculative_impl_draft_dflash::draft` (status
|
||||
6/ABRT, systemd restart, ~503 during reload). DSpark reuses the DFlash
|
||||
block-diffusion machinery, which shares one draft context across
|
||||
parallel slots; concurrent block drafting faults in HIP stream sync.
|
||||
This is the same multi-slot bug class that retired `draft-dflash`
|
||||
(premature EOS). The crash shows on 16k concurrent streams, not on a
|
||||
67k pair — concurrency is the trigger, not size. Fix: PARALLEL=1
|
||||
(requests queue safely). Retired the same day.
|
||||
- **2026-08-15 to 08-16: `--spec-default` (ngram-mod), Vulkan0** — n_match
|
||||
24 / n_min 48 / n_max 64, draftless, PARALLEL=2 (2 x 32k parallel stream
|
||||
test passes). Chosen for parallel safety after DSpark; kept as the
|
||||
fallback under MTP in the current config.
|
||||
|
||||
## Benchmark Method
|
||||
|
||||
The matrix ran 54 configurations: 3 spec types (`draft-mtp,ngram-mod`,
|
||||
@@ -189,37 +133,19 @@ ubatch 256, single slot, on scratch port 7892.
|
||||
|
||||
Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs
|
||||
3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost
|
||||
(users stream tokens), so **Vulkan0** is the production backend. Both
|
||||
`tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s).
|
||||
(users stream tokens). This verdict applies to the standalone
|
||||
configuration; the PP head runs ROCm0 (HIP registers before Vulkan).
|
||||
Both `tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s).
|
||||
High-context decode collapses on both backends versus ~25 t/s at 16k ctx;
|
||||
that is the long-context attention penalty, not a spec artifact.
|
||||
|
||||
## Production Configuration
|
||||
|
||||
`scripts/frank-default-large.env` (live 2026-08-16):
|
||||
|
||||
```ini
|
||||
RPC_SERVERS=192.168.0.235:50052 # Dankserver worker
|
||||
TENSOR_SPLIT=0.40,0.60 # [RPC0, iGPU]
|
||||
CTX_SIZE=401400 # pool = 2 slots x 200700
|
||||
PARALLEL=2
|
||||
SPEC_NONE=0
|
||||
SPEC_DEFAULT=1 # ngram-mod fallback (48/64/24)
|
||||
MTP=1 # draft-mtp, bundled head
|
||||
SPEC_DRAFT_N_MAX=4
|
||||
SPEC_NGRAM_MOD_N_MIN=48
|
||||
SPEC_NGRAM_MOD_N_MAX=64
|
||||
SPEC_NGRAM_MOD_N_MATCH=24
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
BATCH_SIZE=8192
|
||||
UBATCH_SIZE=256
|
||||
```
|
||||
|
||||
There is no `DEVICE` variable. The head must not pass `--device` (the RPC
|
||||
device would be silently dropped). Live slot context: `n_ctx_slot =
|
||||
200704`. The pi `models.json` contextWindow for `frank-default-large`
|
||||
declares 200000, a safe under-estimate of the live slot.
|
||||
The live tunables live in `scripts/frank-default-large.env` (head) and
|
||||
`scripts/frank-pp-worker.env` (worker) in this repo. The Production
|
||||
state section at the top of this page records the current values: Q6_K
|
||||
model + standalone MTP draft, split 0.68/0.32, 262144 ctx, PARALLEL=1,
|
||||
q8_0/f16 KV.
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
Reference in new issue
Block a user