Recording the 54-config matrix, the realistic-prompt correction (n_max=5 not
10), the ubatch findings (256 is the safe/fast point), and the premature-stop
sampling-variance investigation.
Realistic-prompt tuning: n_max=5 peaks ~13.3 t/s (baseline 7.2). matrix used a
repetitive prompt that inflated MTP. n_max=10 is only ~8.6 t/s on real writing.
The f16/f16 KV pool costs 15640 MiB at 800768 cells. At full 800K the
3090 idled at 23966/24576 MiB and the 1472x1472 image warmup graph
failed cudaMalloc (needed +248 MiB). 700416 cells (175104/slot, 4 slots)
leaves ~2.3 GiB free after warmup. Parallel stays 4.
F16 K removes the flash-attn F16-conversion workspace (MMA_F16 prefill
kernel needs no copy when K is already f16). Trades static KV pool size
for the context-proportional workspace growth implicated in the
cublasSgemm_v2 crash class. VERBOSITY=4 run logs real allocation sizes;
refit CTX_SIZE if the pool does not fit on the 24 GiB 3090.
Fix a live crash: cuBLAS cudaErrorInvalidValue in ggml_cuda_mul_mat_cublas
killed the service during a 145k-token prefill burst at ncmoe35/ub3072.
ncmoe 36 moves one more expert layer to CPU: frees ~0.75 GiB peak VRAM
(grid: peak 8606 vs 9360 MiB at ub3072) for -2% prefill / -2% decode.
Keeps prefill priority (top ubatch tier). Docs synced; noted fallback
GGML_CUDA_FORCE_MMQ=1 if pressure persists.
Compiles the 5-subagent research fan-out (llama.cpp RPC, vLLM PP,
bandwidth math, mixed-vendor experiments, Qwen3.6-27B spec methods) into
the KB under docs/Pipeline Parallelism/ with an index, expected
performance estimates (1/2.5/10 GbE), and an EDUCATIONAL-ONLY two-node
Qwen3.6-27B Q8 split script set (MTP/DFlash/ngram-mod variants, DSpark
noted as vLLM-only). Nothing here is deployed or intended to run.
Prefill is the priority for this workload. Switch from decode-max
(ncmoe 32 / ubatch 1024) to prefill-max:
- n-cpu-moe 35, ubatch 3072: prefill ~1458 t/s (top tier), gen ~45.4 t/s,
~1.2 GiB headroom at the 800k pool.
- Gen is independent of ubatch; ub 3072 reaches best prefill with no
decode cost. Docs synced (benchmark Chosen State, deployment, specs).
Retest MTP at n_max 2 (vs 3) for the acceptance/speed tradeoff, keeping
MTP-alone + ubatch 8192. Re-benches gen t/s and re-runs the strict
parallel-safety test.
MTP+ngram benched 12.0-13.6 t/s (vs MTP alone 15.3-19.5) and is
parallel-safe, but slower - layering does not pay off. Back to MTP
alone. Test ubatch 8192 (== batch) using the VRAM headroom; watch for
OOM during load.
Test the MTP+ngram layering (ngram re-drafts on top of the MTP draft) at
ubatch 4096. Must clear the strict parallel-safety test and beat MTP
alone (15.3-19.5 t/s) to stay.
MTP (draft-mtp, n_max 3, a4lg MTP-ONLY-Q8_0) benches 15.3-19.5 t/s vs
ngram-mod 6.7-8.7 t/s and passes the strict LCP-reuse parallel test
(sim 0.996 / f_keep 0.994 launch while another slot generates - the
pattern that killed DFlash). ubatch 4096 to use the ~33 GiB VRAM
headroom; watch peak during load.
ngram-mod benchmark + parallel-safety test passed (A and B both survive
the LCP-reuse launch that killed DFlash). Now benching MTP sidecar
(a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf, draft-mtp, n_max 3).
DFlash causes premature EOS on a generating slot when another task lands
with high LCP prompt reuse (shared draft ctx corruption, reproduced
2026-08-14 3x). Benching ngram-mod vs MTP sidecar to pick the
replacement. serve script now passes --spec-draft-model only for
draft-* spec types.
Cold loads after the cache drop measured 17 minutes on 2026-08-13, longer
than the previous 600s wait. HEALTH_WAIT[dankclaw]=600 iterations (20 min),
dankserver stays at 300 (10 min). AGENTS.md: update cold-load figure.
- deployment.md: document the sudoers NOPASSWD restart helpers on both
targets, the exact-argument matching quirk, and the 2026-08-13
'premature job release' investigation (server never preempts; aborts
were client-side; Dankserver crashed twice under researcher load).
- DFlash/DSpark doc: MTP requires MTP tensors in the target GGUF; the
HauhauCS Qwen3.6-27B GGUF has none (verified tensor listing); MTP-only
draft GGUFs (a4lg/lym00) pair via --spec-draft-model; flag rename
--spec-type mtp -> draft-mtp in b9180.
- AGENTS.md: operational quirks for the helpers, journal abort-vs-release
signature, and the Dankserver cublasSgemm crash signature.
The helper scripts (/usr/local/sbin/frank-restart-<large|small>.sh)
cover unit symlink, cache drop (dankclaw), daemon-reload, and restart
in one root-owned script, run via sudoers NOPASSWD. deploy.sh now uses
the helper when present and falls back to inline sudo commands.
Live-verified on Dankclaw 2026-08-14:
- official rocm/comfyui image is built PYTORCH_ROCM_ARCH=gfx942;gfx950
(datacenter only) -> torch.cuda.is_available() False on gfx1151
- AMD gfx1151-tagged PyTorch build (rocm/vllm:...gfx1151...pytorch_2.9.1)
-> avail True, Radeon 8060S, cap (11,5); needs /dev/kfd + /dev/dri
- torch.backends.vulkan is compiled out of standard wheels; no ComfyUI
--vulkan flag exists; Vulkan is a dead end for ComfyUI
- Lemonade Server does image gen via wrapped stable-diffusion.cpp, whose
HIP kernels include gfx1151 -> works where ComfyUI's torch cannot
New pages: ComfyUI ROCm and Vulkan, Lemonade and sd.cpp; ComfyUI.md
qualified; compose stays --cpu until a gfx1151-enabled torch image is built
- N_CPU_MOE 34→35: 5 expert layers on GPU instead of 6. Frees ~0.4-0.8 GiB
VRAM for OOM safety after high-load sessions OOM'd at 32 and 33.
- Sync stale doc references (deployment.md, Dankserver System Specs.md)
that still claimed 32/8; specs now estimate ~1.8 GiB headroom at 35.
Deterministic deployment instead of agent ritual:
- DROP_CACHES per target (dankclaw=1): runs 'sync; echo 3 >
/proc/sys/vm/drop_caches' before systemctl restart. Without it the
model load stalls at 28 GiB VRAM when the model pages are cached.
- FRANK_SUDO_PW env var: remote sudo pipes it via 'sudo -S' so the
script runs headless. Falls back to plain sudo otherwise.
- Health check runs curl on the target over ssh (127.0.0.1:7770 was
previously checked from the workstation, which can never see it).
Empirically verified on 2026-08-13 (caches dropped): batch 8192 / ubatch
2048 loads and serves at ctx 602100 with 61 GiB VRAM peak. The benchmark
doc's 'ubatch 2048 OOMs' results were confounded by page-cache pressure -
with the model's pages fully cached the KV/graph allocation stalls at
28 GiB VRAM. Drop caches (sync; echo 3 > /proc/sys/vm/drop_caches) before
starting a load to make restarts reliable.
Frank