Merge vllmScripts under llm/ (history-preserving import)
The vllmScripts repository (LLM inference fleet: serve scripts, systemd units, env tunables, deployment runbook, research docs) is merged into this repository. Its full history is grafted under llm/ with a two-parent merge commit so no commit is lost. A follow-up commit reorganizes llm/ into the repo layout: llm/docs -> docs/LLM, llm/scripts -> scripts, llm/old -> old.
This commit is contained in:
commit
918fe4eaec
82 files changed
+10855
No files matched your search
@@ -0,0 +1,88 @@
|
||||
# Head tunables — Qwen3.8-27B pipeline-parallel head (Dankclaw).
|
||||
#
|
||||
# The head is the LOCAL side of the split. It runs llama-server (b10433,
|
||||
# HIP+Vulkan+RPC build in /home/dank/llama.cpp/build) with --rpc pointing at
|
||||
# the worker on Dankserver. The model file lives on this machine only; the
|
||||
# worker receives its tensor share over the network at load time.
|
||||
#
|
||||
# CRITICAL: this script must NOT pass --device. When --device is set,
|
||||
# llama.cpp uses exactly that device list and silently drops the RPC device
|
||||
# (src/llama.cpp llama_prepare_model_devices). Default device selection
|
||||
# inserts RPC servers at the FRONT of the device list, so the split becomes
|
||||
# [RPC0, iGPU] and --tensor-split index 0 is the worker.
|
||||
#
|
||||
# WARNING: do not name any variable here TEMP, HOST, or GPU_MAX_HEAP_SIZE
|
||||
# (HIP runtime collision, see the 2026-08-13 TEMP incident).
|
||||
|
||||
MODEL_DIR=/home/dank/models
|
||||
MODEL_FILE=Qwen3.8-27B-Uncensored-Q6_K.gguf
|
||||
HEAD_BIN=/home/dank/llama.cpp/build/bin/llama-server
|
||||
ALIAS=frank-default-large
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7770
|
||||
|
||||
# Comma-separated RPC endpoints (host:port). One worker here.
|
||||
RPC_SERVERS=192.168.0.235:50052
|
||||
|
||||
# Model split fractions, one per device in order [RPC0, iGPU].
|
||||
# 0.68,0.32 = 45 of 65 blocks on the 3090 (blocks 0-44) - the maximum
|
||||
# for the Q6_K + q8_0/f16 KV config, measured 2026-08-17 phase 2:
|
||||
# - KV per FA layer = 784 MiB (K 1088 B/token + V 2048 B/token @262144)
|
||||
# - weights(f) = 20334 MB layer pool x f (GGUF scan, exact); embd on
|
||||
# the head side (RPC0 buffer observed 10429.88 MiB at f=0.53)
|
||||
# - KV-per-device count = round(f x 66) layers -> 45 layers = 11 FA
|
||||
# (3..43) -> 8624 MiB. 46 layers (f>=0.7045) still 11 FA but margin
|
||||
# drops to ~0.4 GiB; 48 layers (f>=0.7197) adds FA block 47 -> KV
|
||||
# 9408 -> OOM by ~0.85 GiB.
|
||||
# Predicted: 13430 (weights) + 8624 (KV) + ~530 (RS) + 189 (compute) =
|
||||
# ~22773 MiB -> ~0.7 GiB free. Draft model runs on ROCm0 only (measured
|
||||
# phase 1: 1718 MiB, no worker impact).
|
||||
TENSOR_SPLIT=0.68,0.32
|
||||
|
||||
# Context pool. 262144 total = 1 parallel slot x 262144 per slot
|
||||
# (n_ctx_seq = n_ctx / n_seq_max, llama-context.cpp:293).
|
||||
CTX_SIZE=262144
|
||||
PARALLEL=1
|
||||
|
||||
# Speculative decoding. MTP with the STANDALONE draft head (the Q6_K
|
||||
# model has no bundled MTP head; the repo ships the draft sidecar) + the
|
||||
# ngram-mod fallback. Set SPEC_NONE=1 for a raw baseline.
|
||||
SPEC_NONE=0
|
||||
SPEC_DEFAULT=1
|
||||
MTP=1
|
||||
SPEC_DRAFT_N_MAX=4
|
||||
# Standalone MTP draft head (Q8_0 MTP-only sidecar from the same repo).
|
||||
SPEC_DRAFT_MODEL=/home/dank/models/Qwen3.8-27B-Uncensored-draft-Q8_0.gguf
|
||||
SPEC_NGRAM_MOD_N_MIN=48
|
||||
SPEC_NGRAM_MOD_N_MAX=64
|
||||
SPEC_NGRAM_MOD_N_MATCH=24
|
||||
|
||||
# KV on GPU: K q8_0 + V f16 (mismatched, 16 full-attention layers x 4
|
||||
# heads x 256 dim: K = 1088 B/token, V = 2048 B/token -> 3136 B/token ->
|
||||
# 784 MiB per FA layer at 262144, 12.25 GiB total). Mismatched K/V with
|
||||
# Flash Attention requires the GGML_CUDA_FA_ALL_QUANTS rebuild (2026-08-17:
|
||||
# head HIP + worker CUDA at 9b05354ec). Do not revert the binaries without
|
||||
# also reverting this.
|
||||
CACHE_TYPE_K=q8_0
|
||||
CACHE_TYPE_V=f16
|
||||
|
||||
# Batch sizing: production-verified values (2026-08-14).
|
||||
BATCH_SIZE=8192
|
||||
UBATCH_SIZE=256
|
||||
|
||||
# Sampling — Qwen3.8 thinking-mode defaults.
|
||||
TEMPERATURE=1.0
|
||||
TOP_K=20
|
||||
TOP_P=0.95
|
||||
MIN_P=0.0
|
||||
|
||||
THREADS=12
|
||||
THREADS_HTTP=1
|
||||
|
||||
# 4 (info): clean journal for interactive use. Raise to 10 only when
|
||||
# debugging the layer split (the per-layer assignment lines are debug-level).
|
||||
VERBOSITY=4
|
||||
|
||||
# 1: --direct-io model load. Required in practice for large models loaded
|
||||
# over RPC (load hangs otherwise; visorcraft Strix RPC finding).
|
||||
DIRECT_IO=1
|
||||
Reference in new issue
Block a user