docs: llama.cpp stack at production state (2026-08-16) + sibling-project links

- spec matrix: new 'Production state' section — PP head (RPC0+ROCm0,
  split 0.40/0.60), pool 401400 / n_ctx_slot 200704 live, MTP n_max 4
  over ngram-mod fallback (journal acceptance stats), f16 KV.
  2026-08-15 standalone state marked superseded (Vulkan0 verdict was
  standalone-only; PP head runs ROCm0 by default).
- deployment.md: fix frank-default-large section (MTP not 'no draft',
  ROCm0 not Vulkan0, 200704/slot not 262144, f16 not q8_0 KV,
  serve-pp-head.sh not serve-llama-frank-large.sh).
- PP deployment doc: production-state section; test-day open questions
  resolved with journal evidence.
- index.md: NPU line now qwen3.6-moe:35b-a3b @262k ctx; Sibling
  projects section -> serverAppConfigs.
- AGENTS.md: overview reflects PP+MTP reality; sibling pointer.
This commit is contained in:
Frank committed 2026-08-16 23:19:21 -07:00
1 parent f5a78db017
commit 7fe271e65d
5 files changed
+139 -31

No files matched your search

+28 -12
View File
@@ -140,23 +140,36 @@ curl -sf http://127.0.0.1:7770/health
### Dankclaw — frank-default-large
The active service on Dankclaw runs llama.cpp serving the unsloth
Qwen3.8-27B `UD-Q8_K_XL` model under the alias `frank-default-large` on port
7770. It uses llama.cpp's **`--spec-default`** preset (ngram-mod, no draft
model) for speculative decoding on the **Vulkan0** backend, with **2
parallel slots** at 262144 context each, KV cache `q8_0` K / `q8_0` V on
GPU, full layer offload (`--gpu-layers all`, `--fit off`), and batch 8192 /
ubatch 256. The unit file is `scripts/frank-default-large.service`. It
launches through `scripts/serve-llama-frank-large.sh`, which reads tunables
from `scripts/frank-default-large.env`.
The active service on Dankclaw is the llama.cpp **pipeline-parallel
head**. It serves the unsloth Qwen3.8-27B `UD-Q8_K_XL` model under the
alias `frank-default-large` on port 7770. The head splits the tensors
`0.40,0.60` to the `ggml-rpc` worker on Dankserver (RTX 3090, port 50052).
Speculative decoding runs **MTP** (bundled head, `--spec-draft-n-max 4`)
with the `--spec-default` ngram-mod fallback (48/64/24). The head GPU side
runs **ROCm0** (no `--device` flag; default device selection puts RPC
first and picks ROCm for the iGPU). **2 parallel slots** at 200704 context
each (pool 401400), KV cache `f16` K / `f16` V, full layer offload
(`--gpu-layers all`, `--fit off`), batch 8192 / ubatch 256, `--direct-io`.
The unit file is `scripts/frank-default-large.service`. It launches through
`scripts/serve-pp-head.sh`, which reads tunables from
`scripts/frank-default-large.env`.
> Note: 2026-08-16 — the pipeline-parallel split was promoted to
> production as `frank-default-large` (renamed from `frank-pp-head`);
> the standalone single-node service retired. MTP was re-enabled at
> `n_max 4` over the ngram-mod fallback. The pool shrank from 524288 to
> 401400 (2 slots x 200700) to fit the split KV budget. See
> [[Pipeline Parallelism/Qwen3.8 PP Deployment]] for the runbook.
> Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k
> prompts on ROCm, `ROCm error: an illegal memory access`, status 6/ABRT) →
> `--spec-default` (ngram-mod, parallel-safe, PARALLEL=2). The backend went
> ROCm → Vulkan0 after a 262k-ctx bench showed Vulkan decodes 48% faster
> (5.68 vs 3.83 t/s). See [[qwen3.8-spec-matrix]] for both decisions and
> the bench table. Also retired earlier: bundled MTP, the vLLM-ROCm CIRU
> INT4 Ling-3.0-flash runtime, and the llama-serve/vLLM units on Dankclaw.
> (5.68 vs 3.83 t/s). That Vulkan0 verdict applied to the standalone
> configuration; the PP head runs ROCm0 by default (HIP registers before
> Vulkan). See [[qwen3.8-spec-matrix]] for both decisions and the bench
> table. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash
> runtime, and the llama-serve/vLLM units on Dankclaw.
### Dankserver — frank-default-small (STOPPED 2026-08-16)
@@ -216,3 +229,6 @@ This crash class recurs on Dankserver. The 2026-08-14 investigation ruled out ct
- [AGENTS.md](../AGENTS.md) — the git workflow rules
- [README.md](../README.md) — repository overview
- [index.md](index.md) — knowledge base index
- [serverAppConfigs knowledge base](../serverAppConfigs/docs/index.md) —
the sibling Podman app stack on the same hosts; its OpenWebUI consumes
the `:7770` endpoint, and its Gitea instance hosts this repository