docs: llama.cpp stack at production state (2026-08-16) + sibling-project links
- spec matrix: new 'Production state' section — PP head (RPC0+ROCm0, split 0.40/0.60), pool 401400 / n_ctx_slot 200704 live, MTP n_max 4 over ngram-mod fallback (journal acceptance stats), f16 KV. 2026-08-15 standalone state marked superseded (Vulkan0 verdict was standalone-only; PP head runs ROCm0 by default). - deployment.md: fix frank-default-large section (MTP not 'no draft', ROCm0 not Vulkan0, 200704/slot not 262144, f16 not q8_0 KV, serve-pp-head.sh not serve-llama-frank-large.sh). - PP deployment doc: production-state section; test-day open questions resolved with journal evidence. - index.md: NPU line now qwen3.6-moe:35b-a3b @262k ctx; Sibling projects section -> serverAppConfigs. - AGENTS.md: overview reflects PP+MTP reality; sibling pointer.
This commit is contained in:
1 parent
f5a78db017
commit
7fe271e65d
5 files changed
+139
-31
No files matched your search
+28
-12
@@ -140,23 +140,36 @@ curl -sf http://127.0.0.1:7770/health
|
||||
|
||||
### Dankclaw — frank-default-large
|
||||
|
||||
The active service on Dankclaw runs llama.cpp serving the unsloth
|
||||
Qwen3.8-27B `UD-Q8_K_XL` model under the alias `frank-default-large` on port
|
||||
7770. It uses llama.cpp's **`--spec-default`** preset (ngram-mod, no draft
|
||||
model) for speculative decoding on the **Vulkan0** backend, with **2
|
||||
parallel slots** at 262144 context each, KV cache `q8_0` K / `q8_0` V on
|
||||
GPU, full layer offload (`--gpu-layers all`, `--fit off`), and batch 8192 /
|
||||
ubatch 256. The unit file is `scripts/frank-default-large.service`. It
|
||||
launches through `scripts/serve-llama-frank-large.sh`, which reads tunables
|
||||
from `scripts/frank-default-large.env`.
|
||||
The active service on Dankclaw is the llama.cpp **pipeline-parallel
|
||||
head**. It serves the unsloth Qwen3.8-27B `UD-Q8_K_XL` model under the
|
||||
alias `frank-default-large` on port 7770. The head splits the tensors
|
||||
`0.40,0.60` to the `ggml-rpc` worker on Dankserver (RTX 3090, port 50052).
|
||||
Speculative decoding runs **MTP** (bundled head, `--spec-draft-n-max 4`)
|
||||
with the `--spec-default` ngram-mod fallback (48/64/24). The head GPU side
|
||||
runs **ROCm0** (no `--device` flag; default device selection puts RPC
|
||||
first and picks ROCm for the iGPU). **2 parallel slots** at 200704 context
|
||||
each (pool 401400), KV cache `f16` K / `f16` V, full layer offload
|
||||
(`--gpu-layers all`, `--fit off`), batch 8192 / ubatch 256, `--direct-io`.
|
||||
The unit file is `scripts/frank-default-large.service`. It launches through
|
||||
`scripts/serve-pp-head.sh`, which reads tunables from
|
||||
`scripts/frank-default-large.env`.
|
||||
|
||||
> Note: 2026-08-16 — the pipeline-parallel split was promoted to
|
||||
> production as `frank-default-large` (renamed from `frank-pp-head`);
|
||||
> the standalone single-node service retired. MTP was re-enabled at
|
||||
> `n_max 4` over the ngram-mod fallback. The pool shrank from 524288 to
|
||||
> 401400 (2 slots x 200700) to fit the split KV budget. See
|
||||
> [[Pipeline Parallelism/Qwen3.8 PP Deployment]] for the runbook.
|
||||
|
||||
> Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k
|
||||
> prompts on ROCm, `ROCm error: an illegal memory access`, status 6/ABRT) →
|
||||
> `--spec-default` (ngram-mod, parallel-safe, PARALLEL=2). The backend went
|
||||
> ROCm → Vulkan0 after a 262k-ctx bench showed Vulkan decodes 48% faster
|
||||
> (5.68 vs 3.83 t/s). See [[qwen3.8-spec-matrix]] for both decisions and
|
||||
> the bench table. Also retired earlier: bundled MTP, the vLLM-ROCm CIRU
|
||||
> INT4 Ling-3.0-flash runtime, and the llama-serve/vLLM units on Dankclaw.
|
||||
> (5.68 vs 3.83 t/s). That Vulkan0 verdict applied to the standalone
|
||||
> configuration; the PP head runs ROCm0 by default (HIP registers before
|
||||
> Vulkan). See [[qwen3.8-spec-matrix]] for both decisions and the bench
|
||||
> table. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash
|
||||
> runtime, and the llama-serve/vLLM units on Dankclaw.
|
||||
|
||||
### Dankserver — frank-default-small (STOPPED 2026-08-16)
|
||||
|
||||
@@ -216,3 +229,6 @@ This crash class recurs on Dankserver. The 2026-08-14 investigation ruled out ct
|
||||
- [AGENTS.md](../AGENTS.md) — the git workflow rules
|
||||
- [README.md](../README.md) — repository overview
|
||||
- [index.md](index.md) — knowledge base index
|
||||
- [serverAppConfigs knowledge base](../serverAppConfigs/docs/index.md) —
|
||||
the sibling Podman app stack on the same hosts; its OpenWebUI consumes
|
||||
the `:7770` endpoint, and its Gitea instance hosts this repository
|
||||
Reference in new issue
Block a user