docs + script defaults: Q6_K swap, q8_0/f16 KV, split 0.68, FA rebuild
- spec matrix + PP doc + deployment.md + AGENTS.md + index.md: current production = Uncensored Q6_K + standalone MTP draft, mismatched q8_0/f16 KV with FA retained (GGML_CUDA_FA_ALL_QUANTS rebuild at 9b05354ec), split 0.68 (45 blocks), KV 8624/3920, 273 MiB free. - PP doc: 'FA-all-quants rebuild' section (flag, cmake commands, both hosts) + new-model split-limits measurements; old Q8_K_XL limits marked superseded. - serve-pp-head.sh defaults: MODEL_FILE -> Q6_K, CACHE_TYPE_V -> f16 (env already correct; defaults now match production).
This commit is contained in:
1 parent
11f37dafd8
commit
3f88b8e1ed
6 files changed
+112
-50
No files matched your search
@@ -3,12 +3,14 @@
|
||||
## Overview
|
||||
|
||||
This project manages deployment scripts for the LLM inference fleet.
|
||||
Dankclaw runs a llama.cpp pipeline-parallel head for Qwen3.8-27B (unsloth
|
||||
UD-Q8_K_XL): MTP draft (bundled head, `--spec-draft-n-max 4`) over the
|
||||
`--spec-default` ngram-mod fallback, 1 slot at 262144 context (pool
|
||||
262144), tensors split 0.53/0.47 (34 of 65 blocks) to the Dankserver RTX
|
||||
3090 `ggml-rpc` worker, head GPU side on ROCm0 (no `--device` flag), port
|
||||
7770.
|
||||
Dankclaw runs a llama.cpp pipeline-parallel head serving
|
||||
Qwen3.8-27B-Uncensored Q6_K (imatrix): MTP draft (standalone
|
||||
`-draft-Q8_0` sidecar, `--spec-draft-n-max 4`) over the `--spec-default`
|
||||
ngram-mod fallback, 1 slot at 262144 context (pool 262144), tensors
|
||||
split 0.68/0.32 (45 of 65 blocks) to the Dankserver RTX 3090 `ggml-rpc`
|
||||
worker, KV q8_0/f16 (mismatched, FA retained via the
|
||||
GGML_CUDA_FA_ALL_QUANTS rebuild), head GPU side on ROCm0 (no `--device`
|
||||
flag), port 7770.
|
||||
The Dankclaw NPU endpoint (FastFlowLM, `frank-default-small`, port 52625)
|
||||
serves Qwythos-9B-Claude-Mythos-5-1M (abliterated) as registry tag
|
||||
`qwythos:9b` at 262144 ctx (swapped in 2026-08-17; previously
|
||||
|
||||
Reference in new issue
Block a user