docs: llama.cpp stack at production state (2026-08-16) + sibling-project links
- spec matrix: new 'Production state' section — PP head (RPC0+ROCm0, split 0.40/0.60), pool 401400 / n_ctx_slot 200704 live, MTP n_max 4 over ngram-mod fallback (journal acceptance stats), f16 KV. 2026-08-15 standalone state marked superseded (Vulkan0 verdict was standalone-only; PP head runs ROCm0 by default). - deployment.md: fix frank-default-large section (MTP not 'no draft', ROCm0 not Vulkan0, 200704/slot not 262144, f16 not q8_0 KV, serve-pp-head.sh not serve-llama-frank-large.sh). - PP deployment doc: production-state section; test-day open questions resolved with journal evidence. - index.md: NPU line now qwen3.6-moe:35b-a3b @262k ctx; Sibling projects section -> serverAppConfigs. - AGENTS.md: overview reflects PP+MTP reality; sibling pointer.
This commit is contained in:
1 parent
f5a78db017
commit
7fe271e65d
5 files changed
+139
-31
No files matched your search
@@ -2,7 +2,25 @@
|
||||
|
||||
## Overview
|
||||
|
||||
This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.8-27B (unsloth UD-Q8_K_XL) with llama.cpp's `--spec-default` (ngram-mod, no draft model, Vulkan0 backend, 2 parallel slots, port 7770). Dankclaw's NPU endpoint (FastFlowLM, `frank-default-small`, port 52625) serves Qwen3.6-35B-A3B as registry tag `qwen3.6-moe:35b-a3b` at 262144 ctx (swapped in 2026-08-16; the old llama.cpp 35B service on Dankserver is stopped). See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the spec decision (--spec-default + Vulkan0, 2026-08-15; DSpark retired same day for a ROCm parallel crash; DFlash retired earlier for premature EOS).
|
||||
This project manages deployment scripts for the LLM inference fleet.
|
||||
Dankclaw runs a llama.cpp pipeline-parallel head for Qwen3.8-27B (unsloth
|
||||
UD-Q8_K_XL): MTP draft (bundled head, `--spec-draft-n-max 4`) over the
|
||||
`--spec-default` ngram-mod fallback, 2 parallel slots at 200704 context
|
||||
(pool 401400), tensors split 0.40/0.60 to the Dankserver RTX 3090
|
||||
`ggml-rpc` worker, head GPU side on ROCm0 (no `--device` flag), port 7770.
|
||||
The Dankclaw NPU endpoint (FastFlowLM, `frank-default-small`, port 52625)
|
||||
serves Qwen3.6-35B-A3B as registry tag `qwen3.6-moe:35b-a3b` at 262144 ctx
|
||||
(swapped in 2026-08-16; the old llama.cpp 35B service on Dankserver is
|
||||
stopped). See [docs/deployment.md](docs/deployment.md) for the deployment
|
||||
process, [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the
|
||||
spec history and current production state, and
|
||||
[docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md](docs/Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md)
|
||||
for the PP runbook.
|
||||
|
||||
**Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the
|
||||
Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its
|
||||
OpenWebUI consumes the `:7770` endpoint; its Gitea instance hosts this
|
||||
repository. Cross-links live in both knowledge-base indexes.
|
||||
|
||||
## Benchmarking Policy (MANDATORY)
|
||||
|
||||
|
||||
Reference in new issue
Block a user