docs: llama.cpp stack at production state (2026-08-16) + sibling-project links
- spec matrix: new 'Production state' section — PP head (RPC0+ROCm0, split 0.40/0.60), pool 401400 / n_ctx_slot 200704 live, MTP n_max 4 over ngram-mod fallback (journal acceptance stats), f16 KV. 2026-08-15 standalone state marked superseded (Vulkan0 verdict was standalone-only; PP head runs ROCm0 by default). - deployment.md: fix frank-default-large section (MTP not 'no draft', ROCm0 not Vulkan0, 200704/slot not 262144, f16 not q8_0 KV, serve-pp-head.sh not serve-llama-frank-large.sh). - PP deployment doc: production-state section; test-day open questions resolved with journal evidence. - index.md: NPU line now qwen3.6-moe:35b-a3b @262k ctx; Sibling projects section -> serverAppConfigs. - AGENTS.md: overview reflects PP+MTP reality; sibling pointer.
This commit is contained in:
1 parent
f5a78db017
commit
7fe271e65d
5 files changed
+139
-31
No files matched your search
+13
-5
@@ -1,6 +1,8 @@
|
||||
# vLLM Scripts — Knowledge Base
|
||||
|
||||
Documentation for running vLLM inference on two local machines.
|
||||
Documentation for the LLM inference fleet on two local machines:
|
||||
llama.cpp production (PP head + RPC worker), the FastFlowLM NPU endpoint,
|
||||
plus a vLLM research archive.
|
||||
|
||||
## Deployment Topology
|
||||
|
||||
@@ -22,8 +24,8 @@ split 0.40/0.60 to worker, MTP+ngram-mod]
|
||||
D --> E --> F
|
||||
G[FastFlowLM NPU XDNA2 50 TOPS
|
||||
frank-default-small, port 52625]
|
||||
H[gemma4-it:e4b
|
||||
12.2 t/s, 32k ctx, vision]
|
||||
H[qwen3.6-moe:35b-a3b
|
||||
12.5-13.1 t/s, 262k ctx, vision]
|
||||
D --> G --> H
|
||||
end
|
||||
|
||||
@@ -56,7 +58,7 @@ frank-default-small, port 52625]
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU
|
||||
- [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug)
|
||||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec matrix; production = --spec-default (ngram-mod), PARALLEL=2, Vulkan0 backend (2026-08-15)
|
||||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec history + current production state: PP head, MTP (n_max 4) over the ngram-mod fallback, 2 slots @ 200704 (2026-08-16)
|
||||
- [[qwen3.8-strix-optimizations]] — Qwen3.8-27B Strix speed research: ranked levers that keep UD-Q8_K_XL (PP to 3090, MTP tuning, same-bit Q8_0 A/B, prefill path, OS), quant landscape, 2026-08-16 test matrix
|
||||
- [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state
|
||||
- [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small
|
||||
@@ -77,4 +79,10 @@ frank-default-small, port 52625]
|
||||
- [[Pipeline Parallelism/Mixed-Vendor Experiments]] — NVIDIA + Strix Halo real-world runs
|
||||
- [[Pipeline Parallelism/Expected Performance]] — estimated tok/s over 1/2.5/10 GbE
|
||||
- [[Pipeline Parallelism/Sample Script Set]] — educational two-node Qwen3.6-27B Q8 split
|
||||
- [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — production design + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/
|
||||
- [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — the production deployment (promoted 2026-08-16) + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/
|
||||
|
||||
## Sibling projects
|
||||
|
||||
- [serverAppConfigs](../serverAppConfigs/docs/index.md) — the Podman app
|
||||
stack on the same two hosts. Its OpenWebUI consumes the `:7770`
|
||||
endpoint; its Gitea instance hosts this repository.
|
||||
Reference in new issue
Block a user