The f16/f16 KV pool costs 15640 MiB at 800768 cells. At full 800K the
3090 idled at 23966/24576 MiB and the 1472x1472 image warmup graph
failed cudaMalloc (needed +248 MiB). 700416 cells (175104/slot, 4 slots)
leaves ~2.3 GiB free after warmup. Parallel stays 4.
F16 K removes the flash-attn F16-conversion workspace (MMA_F16 prefill
kernel needs no copy when K is already f16). Trades static KV pool size
for the context-proportional workspace growth implicated in the
cublasSgemm_v2 crash class. VERBOSITY=4 run logs real allocation sizes;
refit CTX_SIZE if the pool does not fit on the 24 GiB 3090.
Fix a live crash: cuBLAS cudaErrorInvalidValue in ggml_cuda_mul_mat_cublas
killed the service during a 145k-token prefill burst at ncmoe35/ub3072.
ncmoe 36 moves one more expert layer to CPU: frees ~0.75 GiB peak VRAM
(grid: peak 8606 vs 9360 MiB at ub3072) for -2% prefill / -2% decode.
Keeps prefill priority (top ubatch tier). Docs synced; noted fallback
GGML_CUDA_FORCE_MMQ=1 if pressure persists.
Prefill is the priority for this workload. Switch from decode-max
(ncmoe 32 / ubatch 1024) to prefill-max:
- n-cpu-moe 35, ubatch 3072: prefill ~1458 t/s (top tier), gen ~45.4 t/s,
~1.2 GiB headroom at the 800k pool.
- Gen is independent of ubatch; ub 3072 reaches best prefill with no
decode cost. Docs synced (benchmark Chosen State, deployment, specs).
- N_CPU_MOE 34→35: 5 expert layers on GPU instead of 6. Frees ~0.4-0.8 GiB
VRAM for OOM safety after high-load sessions OOM'd at 32 and 33.
- Sync stale doc references (deployment.md, Dankserver System Specs.md)
that still claimed 32/8; specs now estimate ~1.8 GiB headroom at 35.
The ROCm HIP runtime (libamdhip64) reads the TEMP environment variable as
its JIT kernel-cache temp directory. Commit db3eb01 added TEMP=1.0 to the
env files as a sampling default; the serve scripts export it via set -a.
HIP then used the path "1.0" for its cache, failed, and segfaulted with a
NULL-refcount unref (offset 0x446153 in libamdhip64) during model tensor
upload (ggml_backend_cuda_buffer_set_tensor). This crashed frank-default-
large on every restart from Aug 13 16:22 onward; the earlier c984a3e
"fix" (KV/batch changes) was unrelated.
Rename the sampling variable TEMP -> TEMPERATURE in both env files and
serve scripts. llama-server still receives --temp from the renamed var, so
sampling defaults are unchanged. The HIP runtime no longer sees a bogus
TEMP value. Affects frank-default-large (dankclaw) and frank-default-small
(dankserver).
- frank-default-large: add TEMP=1.0, TOP_K=20, TOP_P=0.95, MIN_P=0.0
(per HauhauCS Qwen3.6-27B model card, thinking-general mode)
- frank-default-small: add same sampling defaults
(per HauhauCS Qwen3.6-35B-A3B model card)
- frank-default-small: reduce N_CPU_MOE from 33 to 34 (6 expert
layers on GPU instead of 7) to prevent OOM during high-load sessions
- Both serve scripts pass --temp, --top-k, --top-p, --min-p to
llama-server; clients can still override per-request via API
- PARALLEL=4 (800000 total ctx), batch=8192, ubatch=2048
- n-cpu-moe=32 (moved 8 expert layers back to GPU)
- ctx-checkpoints=4 for headroom during larger batch compute
- All other tunables unchanged (ngram-mod, no-mmap, no-kv-offload, q8_0/f16 KV)
Lower each stream to 200000 ctx (pool 600000) to free KV VRAM for the larger
batch. Add --no-mmap so the CPU-resident MoE experts load eagerly into RAM
(avoids decode page faults). Keep KV on GPU. Experts already fully on CPU.
Bigger logical/physical batches for prefill throughput. Enable
--ctx-checkpoints to park attention checkpoints in RAM, holding peak GPU
memory down so the larger ubatch fits the 24GB VRAM alongside 3x262144 KV
without OOM. Weight split unchanged (experts CPU, non-experts GPU).
llama.cpp splits --ctx-size across --parallel slots, so 262144 total gave
87552 per stream. Non-negotiable is 262144 per stream: total pool must be
786432.
llama.cpp service replacing retired vllm-awq-qwen36 on port 7770 with alias
frank-default-small. Wrapper sources frank-default-small.env for tunables:
3 parallel slots at 262144 ctx, draft-dflash n-max 7, KV K=q8_0 V=f16 on GPU,
all MoE experts on CPU (--n-cpu-moe 40), non-expert tensors on GPU, mmproj
vision + jinja. Unit: frank-default-small.service (ExecStart in dankserver
repo clone /home/dank/vLLM/scripts).
Frank