docs: llama.cpp stack at production state (2026-08-16) + sibling-project links
- spec matrix: new 'Production state' section — PP head (RPC0+ROCm0, split 0.40/0.60), pool 401400 / n_ctx_slot 200704 live, MTP n_max 4 over ngram-mod fallback (journal acceptance stats), f16 KV. 2026-08-15 standalone state marked superseded (Vulkan0 verdict was standalone-only; PP head runs ROCm0 by default). - deployment.md: fix frank-default-large section (MTP not 'no draft', ROCm0 not Vulkan0, 200704/slot not 262144, f16 not q8_0 KV, serve-pp-head.sh not serve-llama-frank-large.sh). - PP deployment doc: production-state section; test-day open questions resolved with journal evidence. - index.md: NPU line now qwen3.6-moe:35b-a3b @262k ctx; Sibling projects section -> serverAppConfigs. - AGENTS.md: overview reflects PP+MTP reality; sibling pointer.
This commit is contained in:
1 parent
f5a78db017
commit
7fe271e65d
5 files changed
+139
-31
No files matched your search
+52
-13
@@ -1,10 +1,41 @@
|
||||
# Qwen3.8-27B UD-Q8_K_XL Speculative-Decoding Matrix
|
||||
|
||||
This page records the speculative-decoding benchmark and the production
|
||||
tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service on Dankclaw
|
||||
(`frank-default-large`).
|
||||
tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service
|
||||
(`frank-default-large`), now a pipeline-parallel head with a Dankserver
|
||||
worker.
|
||||
|
||||
## Result (applied, 2026-08-15: --spec-default)
|
||||
## Production state (2026-08-16: PP split + MTP)
|
||||
|
||||
Current production, verified against the live process and journal
|
||||
(2026-08-16, service start 15:03). Topology detail:
|
||||
[[Pipeline Parallelism/Qwen3.8 PP Deployment]].
|
||||
|
||||
- Devices: `RPC0` (Dankserver RTX 3090) + `ROCm0` (Strix iGPU). The head
|
||||
passes no `--device` flag. Default selection puts RPC first and picks
|
||||
ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays
|
||||
an option via `HEAD_BIN=.../build-vulkan-rpc`.
|
||||
- Split: `--tensor-split 0.40,0.60` — 40% of the layers on the 3090
|
||||
(its maximum under the KV budget), 60% on the iGPU.
|
||||
- Context: pool 401400, 2 slots. The journal reports
|
||||
`n_ctx_slot = 200704`. The pi `models.json` declares 200000
|
||||
(conservative).
|
||||
- Spec: **MTP re-enabled** — `--spec-type draft-mtp --spec-draft-n-max 4`
|
||||
(bundled head) with the `--spec-default` ngram-mod fallback (48/64/24).
|
||||
Journal acceptance over one production window: MTP mean len 3.00
|
||||
(per-position 0.730/0.541/0.411/0.323, 58934 drafts); ngram-mod as the
|
||||
rare fallback (423 drafts in the same window).
|
||||
- KV: `f16` K / `f16` V on both devices (15680 MiB ROCm0 + 9408 MiB RPC0
|
||||
at load).
|
||||
- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`,
|
||||
`--gpu-layers all`.
|
||||
|
||||
## Result (superseded 2026-08-16: standalone --spec-default)
|
||||
|
||||
Superseded for topology on 2026-08-16, when the PP split became
|
||||
production. The ngram-mod preset survives as the fallback layer under
|
||||
MTP. The Vulkan0 verdict in this section applies to the standalone
|
||||
configuration only.
|
||||
|
||||
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
|
||||
- Spec: llama.cpp **`--spec-default`** preset = **ngram-mod** (n_match 24,
|
||||
@@ -153,22 +184,30 @@ that is the long-context attention penalty, not a spec artifact.
|
||||
|
||||
## Production Configuration
|
||||
|
||||
`scripts/frank-default-large.env`:
|
||||
`scripts/frank-default-large.env` (live 2026-08-16):
|
||||
|
||||
```ini
|
||||
SPEC_DEFAULT=1 # llama.cpp preset = ngram-mod (48/64/24)
|
||||
SPEC_TYPE=
|
||||
DRAFT_FILE=
|
||||
RPC_SERVERS=192.168.0.235:50052 # Dankserver worker
|
||||
TENSOR_SPLIT=0.40,0.60 # [RPC0, iGPU]
|
||||
CTX_SIZE=401400 # pool = 2 slots x 200700
|
||||
PARALLEL=2
|
||||
SPEC_NONE=0
|
||||
SPEC_DEFAULT=1 # ngram-mod fallback (48/64/24)
|
||||
MTP=1 # draft-mtp, bundled head
|
||||
SPEC_DRAFT_N_MAX=4
|
||||
SPEC_NGRAM_MOD_N_MIN=48
|
||||
SPEC_NGRAM_MOD_N_MAX=64
|
||||
SPEC_NGRAM_MOD_N_MATCH=24
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
BATCH_SIZE=8192
|
||||
UBATCH_SIZE=256
|
||||
CTX_SIZE=524288
|
||||
PARALLEL=2 # passes 2x32k parallel stream test (2026-08-15)
|
||||
DEVICE=Vulkan0 # 48% faster decode at 262k ctx
|
||||
```
|
||||
|
||||
Window: computed at 262144 per slot (the pi `models.json` contextWindow for
|
||||
`frank-default-large`). The unit reports `n_ctx_slot = 262144`; CTX_SIZE is
|
||||
a pool ceiling and per-slot stays 262144 regardless of the slot count.
|
||||
There is no `DEVICE` variable. The head must not pass `--device` (the RPC
|
||||
device would be silently dropped). Live slot context: `n_ctx_slot =
|
||||
200704`. The pi `models.json` contextWindow for `frank-default-large`
|
||||
declares 200000, a safe under-estimate of the live slot.
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
Reference in new issue
Block a user