User request 2026-08-19 (second pass): parallel slots degraded
performance. PARALLEL=1, CTX_SIZE=262144 (native max, -c equals the
per-slot context with one slot). Concurrent requests queue on the
single slot; the server never preempts a running generation.
pi models.json contextWindow/maxTokens back to 262144 (outside repo).
Docs updated to match.
User request 2026-08-19: lower per-stream context from 262144 to 200000.
CTX_SIZE is the total; llama.cpp splits it across the 2 parallel slots.
f16 KV drops from ~32 GiB to ~24 GiB, adding OOM headroom.
Docs: sync the 2026-08-18 service swap (frank-default-large on dankclaw,
frank-default-small on dankserver) across AGENTS.md, README.md,
deployment.md, ciru-activefpx-promptforge.md, and index.md. The pi
models.json contextWindow/maxTokens for frank-default-large move
262144 -> 200000 (separate file, outside the repo).
llama.cpp splits -c across -np slots; to give each of the 2 slots the full
262144 native context, set CTX_SIZE=524288. KV ~32 GiB; fits the 96 GiB
unified pool. Update env default, serve-script fallback, and the deployment
doc line.
- Restore the pre-2026-08-17 Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive
Q6_K_P + ngram-mod setup as frank-default-small on Dankserver (port 7770,
upstream llama.cpp b10121, MoE experts on CPU via --n-cpu-moe 36).
- Rename the CIRU Qwen3.8-27B ActiveFPX setup from frank-default-small to
frank-default-large (Dankclaw, port 52625); bump to CTX_SIZE=262144
(native max) and PARALLEL=2 (96 GiB unified pool).
- Retire the Qwen3.8-27B IQ4_XS FastMTP frank-default-large (Dankserver) to
.AGENTTRASH.
- deploy.sh: SERVICE[dankserver]=frank-default-small,
SERVICE[dankclaw]=frank-default-large; HEALTH_WAIT[dankserver]=600.
- frank-default-large: GGML_CUDA_FA_ALL_QUANTS=ON is mandatory in the
FastMTP build; without it q8_0 KV + FA falls back to a slow path and
prefill collapses ~12x (95 -> 1312 t/s at 8K, measured)
- frank-default-small: FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE + pure-python
flash_attn (triton_amd) + stub; without it the vision encoder warmup
OOMs via torch SDPA
- MODEL_FILE -> Qwen3.8-27B-Uncensored-Q6_K.gguf (22.43 GB, no bundled
MTP head; repo ships a standalone draft sidecar)
- SPEC_DRAFT_MODEL -> Qwen3.8-27B-Uncensored-draft-Q8_0.gguf (3.16 GB)
- CACHE_TYPE_K=q8_0, CACHE_TYPE_V=f16 (mismatched, FA retained via the
GGML_CUDA_FA_ALL_QUANTS rebuild of head HIP + worker CUDA at 9b05354ec)
- Split kept at 0.53 for phase 1 (Q6_K frees ~4.7 GiB on the worker at
this split); re-maximize after measuring draft placement
- KV math: K 1088 B/token + V 2048 B/token = 784 MiB per FA layer
0.54 measured: weight boundary moved to 35 blocks but the KV-per-device
count (round(f x 66), independent of weight boundary) flipped to 9 FA ->
KV 9216 MiB -> OOM by ~740 MiB. The flip sits at f ~0.538, just below
the weight threshold for block 35 -> the 35-block config is unreachable.
0.53 gives 34 blocks (0-33), 8 FA, KV 8192 MiB under every rounding rule
(round/floor/ceil of 0.53x66 = 34.98 -> 34 or 35, never 36), predicted
~1.1 GiB free.
Single full-context slot instead of two half-context slots. The 3090 takes
~33 of 66 layers (up from 26): the VRAM budget frees up because the f16 KV
pool shrinks 401400 -> 262144. 33 layers is the max with a safety margin
(34 -> 244 MiB headroom, 35 -> OOM). Verified math: /tmp/vram-math.py from
live journal buffers.
Realistic-prompt tuning: n_max=5 peaks ~13.3 t/s (baseline 7.2). matrix used a
repetitive prompt that inflated MTP. n_max=10 is only ~8.6 t/s on real writing.
Retest MTP at n_max 2 (vs 3) for the acceptance/speed tradeoff, keeping
MTP-alone + ubatch 8192. Re-benches gen t/s and re-runs the strict
parallel-safety test.
MTP+ngram benched 12.0-13.6 t/s (vs MTP alone 15.3-19.5) and is
parallel-safe, but slower - layering does not pay off. Back to MTP
alone. Test ubatch 8192 (== batch) using the VRAM headroom; watch for
OOM during load.
Test the MTP+ngram layering (ngram re-drafts on top of the MTP draft) at
ubatch 4096. Must clear the strict parallel-safety test and beat MTP
alone (15.3-19.5 t/s) to stay.
MTP (draft-mtp, n_max 3, a4lg MTP-ONLY-Q8_0) benches 15.3-19.5 t/s vs
ngram-mod 6.7-8.7 t/s and passes the strict LCP-reuse parallel test
(sim 0.996 / f_keep 0.994 launch while another slot generates - the
pattern that killed DFlash). ubatch 4096 to use the ~33 GiB VRAM
headroom; watch peak during load.
ngram-mod benchmark + parallel-safety test passed (A and B both survive
the LCP-reuse launch that killed DFlash). Now benching MTP sidecar
(a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf, draft-mtp, n_max 3).
DFlash causes premature EOS on a generating slot when another task lands
with high LCP prompt reuse (shared draft ctx corruption, reproduced
2026-08-14 3x). Benching ngram-mod vs MTP sidecar to pick the
replacement. serve script now passes --spec-draft-model only for
draft-* spec types.
Empirically verified on 2026-08-13 (caches dropped): batch 8192 / ubatch
2048 loads and serves at ctx 602100 with 61 GiB VRAM peak. The benchmark
doc's 'ubatch 2048 OOMs' results were confounded by page-cache pressure -
with the model's pages fully cached the KV/graph allocation stalls at
28 GiB VRAM. Drop caches (sync; echo 3 > /proc/sys/vm/drop_caches) before
starting a load to make restarts reliable.
The ROCm HIP runtime (libamdhip64) reads the TEMP environment variable as
its JIT kernel-cache temp directory. Commit db3eb01 added TEMP=1.0 to the
env files as a sampling default; the serve scripts export it via set -a.
HIP then used the path "1.0" for its cache, failed, and segfaulted with a
NULL-refcount unref (offset 0x446153 in libamdhip64) during model tensor
upload (ggml_backend_cuda_buffer_set_tensor). This crashed frank-default-
large on every restart from Aug 13 16:22 onward; the earlier c984a3e
"fix" (KV/batch changes) was unrelated.
Rename the sampling variable TEMP -> TEMPERATURE in both env files and
serve scripts. llama-server still receives --temp from the renamed var, so
sampling defaults are unchanged. The HIP runtime no longer sees a bogus
TEMP value. Affects frank-default-large (dankclaw) and frank-default-small
(dankserver).
The Aug 13 change crashed frank-default-large in a restart loop. The service
segfaulted ~15s into model load (SIGSEGV in hipMalloc during load). The
benchmark (frank-default-large-benchmark.md) already proved ubatch 2048 OOMs
(40.9GB prefill buffer) and batch 2048 / ubatch 512 is the fits-and-loads
config on the 96GiB UMA budget. Restore q8_0/q8_0 KV cache (FA pairing) and
batch 2048 / ubatch 512.
Also fix deployment.md: KV is q8_0 K / q8_0 V (not f16 V) since 334b6af.
- Remove CACHE_TYPE_K/CACHE_TYPE_V (fp16 default on GPU)
- Increase BATCH_SIZE from 2048 to 8192
- Increase UBATCH_SIZE from 512 to 2048
- Remove --cache-type-k and --cache-type-v from llama-server flags
- frank-default-large: add TEMP=1.0, TOP_K=20, TOP_P=0.95, MIN_P=0.0
(per HauhauCS Qwen3.6-27B model card, thinking-general mode)
- frank-default-small: add same sampling defaults
(per HauhauCS Qwen3.6-35B-A3B model card)
- frank-default-small: reduce N_CPU_MOE from 33 to 34 (6 expert
layers on GPU instead of 7) to prevent OOM during high-load sessions
- Both serve scripts pass --temp, --top-k, --top-p, --min-p to
llama-server; clients can still override per-request via API
Root cause of ~1 t/s long-context decode: llama.cpp build has
GGML_CUDA_FA_ALL_QUANTS=OFF, and FA requires K type == V type.
Mixed cache q8_0/f16 silently disabled FA, collapsing decode at
200k ctx. Set V to q8_0 (halves V memory too) and retune
threads 12/1 for 16 Zen5 cores.
Frank