- N_CPU_MOE 34→35: 5 expert layers on GPU instead of 6. Frees ~0.4-0.8 GiB
VRAM for OOM safety after high-load sessions OOM'd at 32 and 33.
- Sync stale doc references (deployment.md, Dankserver System Specs.md)
that still claimed 32/8; specs now estimate ~1.8 GiB headroom at 35.
The ROCm HIP runtime (libamdhip64) reads the TEMP environment variable as
its JIT kernel-cache temp directory. Commit db3eb01 added TEMP=1.0 to the
env files as a sampling default; the serve scripts export it via set -a.
HIP then used the path "1.0" for its cache, failed, and segfaulted with a
NULL-refcount unref (offset 0x446153 in libamdhip64) during model tensor
upload (ggml_backend_cuda_buffer_set_tensor). This crashed frank-default-
large on every restart from Aug 13 16:22 onward; the earlier c984a3e
"fix" (KV/batch changes) was unrelated.
Rename the sampling variable TEMP -> TEMPERATURE in both env files and
serve scripts. llama-server still receives --temp from the renamed var, so
sampling defaults are unchanged. The HIP runtime no longer sees a bogus
TEMP value. Affects frank-default-large (dankclaw) and frank-default-small
(dankserver).
- frank-default-large: add TEMP=1.0, TOP_K=20, TOP_P=0.95, MIN_P=0.0
(per HauhauCS Qwen3.6-27B model card, thinking-general mode)
- frank-default-small: add same sampling defaults
(per HauhauCS Qwen3.6-35B-A3B model card)
- frank-default-small: reduce N_CPU_MOE from 33 to 34 (6 expert
layers on GPU instead of 7) to prevent OOM during high-load sessions
- Both serve scripts pass --temp, --top-k, --top-p, --min-p to
llama-server; clients can still override per-request via API
- PARALLEL=4 (800000 total ctx), batch=8192, ubatch=2048
- n-cpu-moe=32 (moved 8 expert layers back to GPU)
- ctx-checkpoints=4 for headroom during larger batch compute
- All other tunables unchanged (ngram-mod, no-mmap, no-kv-offload, q8_0/f16 KV)
Lower each stream to 200000 ctx (pool 600000) to free KV VRAM for the larger
batch. Add --no-mmap so the CPU-resident MoE experts load eagerly into RAM
(avoids decode page faults). Keep KV on GPU. Experts already fully on CPU.
Bigger logical/physical batches for prefill throughput. Enable
--ctx-checkpoints to park attention checkpoints in RAM, holding peak GPU
memory down so the larger ubatch fits the 24GB VRAM alongside 3x262144 KV
without OOM. Weight split unchanged (experts CPU, non-experts GPU).
llama.cpp splits --ctx-size across --parallel slots, so 262144 total gave
87552 per stream. Non-negotiable is 262144 per stream: total pool must be
786432.
llama.cpp service replacing retired vllm-awq-qwen36 on port 7770 with alias
frank-default-small. Wrapper sources frank-default-small.env for tunables:
3 parallel slots at 262144 ctx, draft-dflash n-max 7, KV K=q8_0 V=f16 on GPU,
all MoE experts on CPU (--n-cpu-moe 40), non-expert tensors on GPU, mmproj
vision + jinja. Unit: frank-default-small.service (ExecStart in dankserver
repo clone /home/dank/vLLM/scripts).
Frank