dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs + script defaults: Q6_K swap, q8_0/f16 KV, split 0.68, FA rebuild

Browse Source 
- spec matrix + PP doc + deployment.md + AGENTS.md + index.md: current
  production = Uncensored Q6_K + standalone MTP draft, mismatched
  q8_0/f16 KV with FA retained (GGML_CUDA_FA_ALL_QUANTS rebuild at
  9b05354ec), split 0.68 (45 blocks), KV 8624/3920, 273 MiB free.
- PP doc: 'FA-all-quants rebuild' section (flag, cmake commands, both
  hosts) + new-model split-limits measurements; old Q8_K_XL limits
  marked superseded.
- serve-pp-head.sh defaults: MODEL_FILE -> Q6_K, CACHE_TYPE_V -> f16
  (env already correct; defaults now match production).
This commit is contained in:
Frank
2026-08-17 02:08:08 -07:00
parent 11f37dafd8
commit 3f88b8e1ed
6 changed files with 112 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deployment.md
+21 -12
View File
@@ -141,22 +141,31 @@ curl -sf http://127.0.0.1:7770/health
### Dankclaw — frank-default-large
The active service on Dankclaw is the llama.cpp **pipeline-parallel
head**. It serves the unsloth Qwen3.8-27B `UD-Q8_K_XL` model under the
alias `frank-default-large` on port 7770. The head splits the tensors
`0.53,0.47` to the `ggml-rpc` worker on Dankserver (RTX 3090, port 50052):
34 of 65 blocks (0-33) on the 3090 — the maximum provably-safe split
(measured 2026-08-17; see the PP doc for the boundary analysis).
Speculative decoding runs **MTP** (bundled head, `--spec-draft-n-max 4`)
with the `--spec-default` ngram-mod fallback (48/64/24). The head GPU side
runs **ROCm0** (no `--device` flag; default device selection puts RPC
first and picks ROCm for the iGPU). **1 slot** at 262144 context (pool
262144, `n_ctx_slot = 262144`), KV cache `f16` K / `f16` V (8192 MiB per
device), full layer offload (`--gpu-layers all`, `--fit off`), batch 8192
/ ubatch 256, `--direct-io`. The unit file is
head**. It serves `Qwen3.8-27B-Uncensored-Q6_K.gguf` (JonathanColetti,
imatrix) under the alias `frank-default-large` on port 7770. The head
splits the tensors `0.68,0.32` to the `ggml-rpc` worker on Dankserver
(RTX 3090, port 50052): 45 of 65 blocks (0-44) on the 3090 — the
maximum for this config (measured 2026-08-17; see the PP doc).
Speculative decoding runs **MTP** with a standalone draft head
(`--spec-draft-model`, `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`, n_max
4) plus the `--spec-default` ngram-mod fallback (48/64/24). The head GPU
side runs **ROCm0** (no `--device` flag; default device selection puts
RPC first and picks ROCm for the iGPU). **1 slot** at 262144 context
(pool 262144, `n_ctx_slot = 262144`), KV cache **K q8_0 + V f16**
(mismatched, 784 MiB per FA layer; RPC0 8624 + ROCm0 3920 MiB) with
Flash Attention retained via the GGML_CUDA_FA_ALL_QUANTS rebuild, full
layer offload (`--gpu-layers all`, `--fit off`), batch 8192 / ubatch
256, `--direct-io`. The unit file is
`scripts/frank-default-large.service`. It launches through
`scripts/serve-pp-head.sh`, which reads tunables from
`scripts/frank-default-large.env`.
> Note: 2026-08-17 — the Q6_K swap: model + standalone draft head,
> mismatched q8_0/f16 KV with FA (both binaries rebuilt with
> GGML_CUDA_FA_ALL_QUANTS at 9b05354ec), split 0.68 (was 0.53 with the
> Q8_K_XL). Worker at 23854/24576 MiB, ~273 MiB free. MTP acceptance
> 0.75-0.83 (mean len 4.0-4.3), stronger than the old bundled head.
> Note: 2026-08-17 — single-slot config: `PARALLEL=1`, pool 262144
> (was 2 slots x 200700 at 401400), split 0.53 (was 0.40). The 3090
> landed at 23460/24576 MiB, ~667 MiB free. Same-day incident: a full