docs + script defaults: Q6_K swap, q8_0/f16 KV, split 0.68, FA rebuild

- spec matrix + PP doc + deployment.md + AGENTS.md + index.md: current
  production = Uncensored Q6_K + standalone MTP draft, mismatched
  q8_0/f16 KV with FA retained (GGML_CUDA_FA_ALL_QUANTS rebuild at
  9b05354ec), split 0.68 (45 blocks), KV 8624/3920, 273 MiB free.
- PP doc: 'FA-all-quants rebuild' section (flag, cmake commands, both
  hosts) + new-model split-limits measurements; old Q8_K_XL limits
  marked superseded.
- serve-pp-head.sh defaults: MODEL_FILE -> Q6_K, CACHE_TYPE_V -> f16
  (env already correct; defaults now match production).
This commit is contained in:
Frank
2026-08-17 02:08:08 -07:00
parent 11f37dafd8
commit 3f88b8e1ed
6 changed files with 112 additions and 50 deletions
+2 -2
View File
@@ -25,7 +25,7 @@ fi
# --- defaults (used when the env file does not set them) -------------------
: "${MODEL_DIR:=/home/dank/models}"
: "${MODEL_FILE:=Qwen3.8-27B-UD-Q8_K_XL.gguf}"
: "${MODEL_FILE:=Qwen3.8-27B-Uncensored-Q6_K.gguf}"
: "${HEAD_BIN:=/home/dank/llama.cpp/build/bin/llama-server}"
: "${ALIAS:=frank-pp-large}"
: "${BIND_ADDR:=0.0.0.0}"
@@ -43,7 +43,7 @@ fi
: "${SPEC_NGRAM_MOD_N_MAX:=64}"
: "${SPEC_NGRAM_MOD_N_MATCH:=24}"
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=q8_0}"
: "${CACHE_TYPE_V:=f16}"
: "${BATCH_SIZE:=8192}"
: "${UBATCH_SIZE:=256}"
: "${TEMPERATURE:=1.0}"