docs + script defaults: Q6_K swap, q8_0/f16 KV, split 0.68, FA rebuild

- spec matrix + PP doc + deployment.md + AGENTS.md + index.md: current
  production = Uncensored Q6_K + standalone MTP draft, mismatched
  q8_0/f16 KV with FA retained (GGML_CUDA_FA_ALL_QUANTS rebuild at
  9b05354ec), split 0.68 (45 blocks), KV 8624/3920, 273 MiB free.
- PP doc: 'FA-all-quants rebuild' section (flag, cmake commands, both
  hosts) + new-model split-limits measurements; old Q8_K_XL limits
  marked superseded.
- serve-pp-head.sh defaults: MODEL_FILE -> Q6_K, CACHE_TYPE_V -> f16
  (env already correct; defaults now match production).
This commit is contained in:
Frank committed 2026-08-17 02:08:08 -07:00
1 parent 11f37dafd8
commit 3f88b8e1ed
6 files changed
+112 -50

No files matched your search

+8 -6
View File
@@ -3,12 +3,14 @@
## Overview
This project manages deployment scripts for the LLM inference fleet.
Dankclaw runs a llama.cpp pipeline-parallel head for Qwen3.8-27B (unsloth
UD-Q8_K_XL): MTP draft (bundled head, `--spec-draft-n-max 4`) over the
`--spec-default` ngram-mod fallback, 1 slot at 262144 context (pool
262144), tensors split 0.53/0.47 (34 of 65 blocks) to the Dankserver RTX
3090 `ggml-rpc` worker, head GPU side on ROCm0 (no `--device` flag), port
7770.
Dankclaw runs a llama.cpp pipeline-parallel head serving
Qwen3.8-27B-Uncensored Q6_K (imatrix): MTP draft (standalone
`-draft-Q8_0` sidecar, `--spec-draft-n-max 4`) over the `--spec-default`
ngram-mod fallback, 1 slot at 262144 context (pool 262144), tensors
split 0.68/0.32 (45 of 65 blocks) to the Dankserver RTX 3090 `ggml-rpc`
worker, KV q8_0/f16 (mismatched, FA retained via the
GGML_CUDA_FA_ALL_QUANTS rebuild), head GPU side on ROCm0 (no `--device`
flag), port 7770.
The Dankclaw NPU endpoint (FastFlowLM, `frank-default-small`, port 52625)
serves Qwythos-9B-Claude-Mythos-5-1M (abliterated) as registry tag
`qwythos:9b` at 262144 ctx (swapped in 2026-08-17; previously