- MODEL_FILE -> Qwen3.8-27B-Uncensored-Q6_K.gguf (22.43 GB, no bundled
MTP head; repo ships a standalone draft sidecar)
- SPEC_DRAFT_MODEL -> Qwen3.8-27B-Uncensored-draft-Q8_0.gguf (3.16 GB)
- CACHE_TYPE_K=q8_0, CACHE_TYPE_V=f16 (mismatched, FA retained via the
GGML_CUDA_FA_ALL_QUANTS rebuild of head HIP + worker CUDA at 9b05354ec)
- Split kept at 0.53 for phase 1 (Q6_K frees ~4.7 GiB on the worker at
this split); re-maximize after measuring draft placement
- KV math: K 1088 B/token + V 2048 B/token = 784 MiB per FA layer
- FLM_MODEL=qwythos:9b, CTX 262144, turbo+asr kept
- Add FLM_CONFIG_PATH/FLM_XCLBIN_PATH: the custom tag lives in the user
registry (~/.config/flm/model_list.json) + user xclbins; without them
flm hits the unknown-tag fallback and serves llama3.2:1b silently
- Note FLM 0.9.45 -> 1.0.1 runtime upgrade (pack declares flm_version 1.0.0)
0.54 measured: weight boundary moved to 35 blocks but the KV-per-device
count (round(f x 66), independent of weight boundary) flipped to 9 FA ->
KV 9216 MiB -> OOM by ~740 MiB. The flip sits at f ~0.538, just below
the weight threshold for block 35 -> the 35-block config is unreachable.
0.53 gives 34 blocks (0-33), 8 FA, KV 8192 MiB under every rounding rule
(round/floor/ceil of 0.53x66 = 34.98 -> 34 or 35, never 36), predicted
~1.1 GiB free.
Single full-context slot instead of two half-context slots. The 3090 takes
~33 of 66 layers (up from 26): the VRAM budget frees up because the f16 KV
pool shrinks 401400 -> 262144. 33 layers is the max with a safety margin
(34 -> 244 MiB headroom, 35 -> OOM). Verified math: /tmp/vram-math.py from
live journal buffers.
- scripts: serve-pp-head/worker, envs, systemd units, pp-test.sh orchestrator, pp-bench.py
- head design verified in b10433 source + live 4B smoke test:
* no --device with --rpc (RPC device silently dropped otherwise)
* default device selection = [RPC0, ROCm0] (HIP registers before Vulkan)
* default split proportional to reported free mem = ~20/80 here, pin 1,1
* -dio required for large model over RPC
- docs: qwen3.8-strix-optimizations (ranked no-quant-downgrade levers,
quant landscape, 2026-08-16 test matrix), PP deployment runbook,
index updates, Dankclaw Improvements corrections (Vulkan0, no power_profile sysfs)
- worker built at 9b05354ec in /home/dank/llama-cpp-rpc worktree (CUDA+RPC)
- second head build build-vulkan-rpc (HIP off) for backend A/B
Recording the 54-config matrix, the realistic-prompt correction (n_max=5 not
10), the ubatch findings (256 is the safe/fast point), and the premature-stop
sampling-variance investigation.
Realistic-prompt tuning: n_max=5 peaks ~13.3 t/s (baseline 7.2). matrix used a
repetitive prompt that inflated MTP. n_max=10 is only ~8.6 t/s on real writing.
The f16/f16 KV pool costs 15640 MiB at 800768 cells. At full 800K the
3090 idled at 23966/24576 MiB and the 1472x1472 image warmup graph
failed cudaMalloc (needed +248 MiB). 700416 cells (175104/slot, 4 slots)
leaves ~2.3 GiB free after warmup. Parallel stays 4.
F16 K removes the flash-attn F16-conversion workspace (MMA_F16 prefill
kernel needs no copy when K is already f16). Trades static KV pool size
for the context-proportional workspace growth implicated in the
cublasSgemm_v2 crash class. VERBOSITY=4 run logs real allocation sizes;
refit CTX_SIZE if the pool does not fit on the 24 GiB 3090.
Fix a live crash: cuBLAS cudaErrorInvalidValue in ggml_cuda_mul_mat_cublas
killed the service during a 145k-token prefill burst at ncmoe35/ub3072.
ncmoe 36 moves one more expert layer to CPU: frees ~0.75 GiB peak VRAM
(grid: peak 8606 vs 9360 MiB at ub3072) for -2% prefill / -2% decode.
Keeps prefill priority (top ubatch tier). Docs synced; noted fallback
GGML_CUDA_FORCE_MMQ=1 if pressure persists.
Prefill is the priority for this workload. Switch from decode-max
(ncmoe 32 / ubatch 1024) to prefill-max:
- n-cpu-moe 35, ubatch 3072: prefill ~1458 t/s (top tier), gen ~45.4 t/s,
~1.2 GiB headroom at the 800k pool.
- Gen is independent of ubatch; ub 3072 reaches best prefill with no
decode cost. Docs synced (benchmark Chosen State, deployment, specs).
Retest MTP at n_max 2 (vs 3) for the acceptance/speed tradeoff, keeping
MTP-alone + ubatch 8192. Re-benches gen t/s and re-runs the strict
parallel-safety test.
MTP+ngram benched 12.0-13.6 t/s (vs MTP alone 15.3-19.5) and is
parallel-safe, but slower - layering does not pay off. Back to MTP
alone. Test ubatch 8192 (== batch) using the VRAM headroom; watch for
OOM during load.
Test the MTP+ngram layering (ngram re-drafts on top of the MTP draft) at
ubatch 4096. Must clear the strict parallel-safety test and beat MTP
alone (15.3-19.5 t/s) to stay.
MTP (draft-mtp, n_max 3, a4lg MTP-ONLY-Q8_0) benches 15.3-19.5 t/s vs
ngram-mod 6.7-8.7 t/s and passes the strict LCP-reuse parallel test
(sim 0.996 / f_keep 0.994 launch while another slot generates - the
pattern that killed DFlash). ubatch 4096 to use the ~33 GiB VRAM
headroom; watch peak during load.
Frank