From 8b3cd2ff791a32769e3621e0979215764b3b88cb Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 22 Aug 2026 03:30:58 -0700 Subject: [PATCH] large: batch profile flipped to the runner's qualified 2048/512 (fastest measured prefill, no decode change) --- AGENTS.md | 9 +++++---- docs/kairic-edge.md | 33 +++++++++++++++------------------ scripts/frank-default-large.env | 11 ++++++----- scripts/serve-frank-large.sh | 16 +++++++++------- 4 files changed, 35 insertions(+), 34 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 07aebe1..7797c5a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -138,10 +138,11 @@ RDNA 3.5 native IU4 matrix instruction. The build uses the release script against the private TheRock ROCm 7.15 SDK kept in the retired ling3-ciru venv. The model files (GGUF + FFN/GDN/GDN-Output .pfs, 26.05 GiB) live in `/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge/`. The production profile -adds batch 4096 / ubatch 4096, `--no-mmap`, and reasoning on (deepseek -format, preserve_thinking) on top of the pinned runner. Ubatch 4096 slows -prefill ~25% versus ubatch 2048 with no decode change; see -docs/kairic-edge.md before changing the batch profile. +adds `--no-mmap` and reasoning on (deepseek format, preserve_thinking) on +top of the pinned runner; batch 2048 / ubatch 512 are the runner's +qualified sizes. Benchmarks: ubatch 512 prefill is 60-90% faster than +ubatch 4096 with no decode change; see docs/kairic-edge.md before +changing the batch profile. **Release gap — CK `config.h`:** the pinned CK commit ships only `config.h.in`; the release script and guide never generate diff --git a/docs/kairic-edge.md b/docs/kairic-edge.md index bba2151..743f02f 100644 --- a/docs/kairic-edge.md +++ b/docs/kairic-edge.md @@ -211,7 +211,7 @@ See [[deployment]] for the update process. |---------|-------|--------| | Bind | `0.0.0.0:52625` | Local (runner default `127.0.0.1:8080`) | | Context | 262144, 1 slot | Pinned runner | -| Batch / ubatch | 4096 / 4096 | Local (runner: 2048 / 512; v3: 4096 / 4096) | +| Batch / ubatch | 2048 / 512 | Pinned runner (fastest measured prefill) | | Model loading | `--no-mmap` | Local (benchmarks: no perf difference) | | Threads | 16 target / 32 batch | Pinned runner | | KV | f16 target and draft | Pinned runner | @@ -228,10 +228,6 @@ Local deviations and their reasons: - **Reasoning on + deepseek**: thinking goes to `message.reasoning_content`, matching the retired v3 service. `preserve_thinking=true` keeps thinking tokens in the history across turns. -- **Batch 4096 / ubatch 4096**: larger than the runner; matches the - retired v3 profile. Unified memory has headroom. Ubatch 4096 slows - prefill (~25% versus ubatch 2048, ~50% versus the runner's ubatch 512) - and does not change decode. - **`--no-mmap`**: benchmarks show no performance difference versus mmap. Kept per operator preference (2026-08-22). - **Target greedy argmax fast path off**: the pinned runner sets @@ -272,22 +268,23 @@ Findings: represent prose chat. The card's natural-prose slice measured 34.88 t/s on the certified system. -### ubatch 4096 + --no-mmap profile (2026-08-22) +### Batch profile comparison (2026-08-22) -The production profile moved from batch 4096 / ubatch 2048 with mmap to -batch 4096 / ubatch 4096 with `--no-mmap` (matching the v3 profile). Same -harness, same prompts. +Three batch profiles ran on the identical chat profile (fast path off, +reasoning on, `--no-mmap`; the ub2048 row was the earlier mmap run, and +mmap makes no difference): -| Context | ub2048 decode (cold/warm) | ub4096 decode (cold/warm) | ub2048 prefill cold | ub4096 prefill cold | -|---------|---------------------------|---------------------------|---------------------|---------------------| -| ~8k | 19.79 / 19.36 | 22.65 / 19.78 | 348 | 266 | -| ~16k | 19.83 / 18.75 | 18.13 / 22.48 | 370 | 258 | -| ~64k | 14.61 / - | 14.11 / 13.80 | 232 | 203 | +| Context | b2048/ub512 decode (cold/warm) | b4096/ub2048 decode | b4096/ub4096 decode | ub512 prefill cold | ub4096 prefill cold | +|---------|--------------------------------|---------------------|---------------------|--------------------|---------------------| +| ~8k | 19.6 / 18.7 | 19.8 / 19.4 | 22.7 / 19.8 | 524 | 266 | +| ~16k | 21.8 / 21.5 | 19.8 / 18.8 | 18.1 / 22.5 | 485 | 258 | +| ~64k | 13.7 / 15.7 | 14.6 / - | 14.1 / 13.8 | 323 | 203 | -Ubatch 4096 does not change decode at any depth. It slows prefill by -~25% versus ubatch 2048 and ~50% versus the runner's ubatch 512 -(506 t/s at 8k). The IU4 lane accelerates 96-512-row prompt and -verification shapes; large ubatches do not use those shapes. +Ubatch is a prefill knob. Ubatch 512 gives the fastest prefill at every +depth, 60-90% above ubatch 4096. Decode is identical across all three +profiles within run-to-run noise. The IU4 lane accelerates 96-512-row +prompt and verification shapes; ubatch 512 matches those shapes, large +ubatches do not. Production runs the runner's qualified 2048/512. ## Rollback to v3 diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index cdef6da..6aa2e32 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -50,11 +50,12 @@ LISTEN_PORT=52625 # while the prompt cache is enabled. CTX_SIZE=262144 PARALLEL=1 -# Local override: the pinned runner uses 2048/512; production chat uses a -# larger 4096/4096 batch (unified memory has headroom; matches the retired -# v3 profile). -BATCH_SIZE=4096 -UBATCH_SIZE=4096 +# Batch profile: the pinned runner's qualified sizes (2048/512). Benchmarks +# on 2026-08-22: ubatch 512 gives the fastest prefill at every depth +# (524/485/323 t/s at 8k/16k/64k vs 266/258/203 for ubatch 4096) with no +# decode difference. Do not raise ubatch without re-benching prefill. +BATCH_SIZE=2048 +UBATCH_SIZE=512 TARGET_THREADS=16 BATCH_THREADS=32 CACHE_TYPE_K=f16 diff --git a/scripts/serve-frank-large.sh b/scripts/serve-frank-large.sh index 9d9a7ae..2ad9d1f 100755 --- a/scripts/serve-frank-large.sh +++ b/scripts/serve-frank-large.sh @@ -16,9 +16,10 @@ # differences: --host/--port come from BIND_ADDR/LISTEN_PORT (safe names; # the ROCm HIP runtime reads a bare HOST env var, see the 2026-08-13 TEMP # incident), LD_LIBRARY_PATH uses this machine's split 7.15 SDK layout, -# the profile runs --no-mmap, batch 4096 / ubatch 4096, reasoning on with -# deepseek format and preserve_thinking, and the target greedy argmax fast -# path is disabled for non-greedy chat sampling. +# the profile runs --no-mmap, reasoning on with deepseek format and +# preserve_thinking, and the target greedy argmax fast path is disabled +# for non-greedy chat sampling. Batch 2048 / ubatch 512 are the runner's +# qualified sizes. # # Tunables come from frank-default-large.env in this directory unless # FRANK_LARGE_ENV_FILE overrides that path. @@ -49,8 +50,8 @@ fi : "${LISTEN_PORT:=52625}" : "${CTX_SIZE:=262144}" : "${PARALLEL:=1}" -: "${BATCH_SIZE:=4096}" -: "${UBATCH_SIZE:=4096}" +: "${BATCH_SIZE:=2048}" +: "${UBATCH_SIZE:=512}" : "${TARGET_THREADS:=16}" : "${BATCH_THREADS:=32}" : "${CACHE_TYPE_K:=f16}" @@ -125,10 +126,11 @@ export PROMPTFORGE_ENABLE_IU4_DECODE_M2_M5=0 export PROMPTFORGE_OUTPUT_K8_STRICT_GREEDY=0 # Pinned runner flags (qualified 262K one-slot profile, native MTP depth -# 4, deterministic sampler). Local overrides for production chat: +# 4, deterministic sampler; batch 2048 / ubatch 512 are the runner's +# qualified sizes and give the fastest measured prefill). Local overrides +# for production chat: # - reasoning on + deepseek format (thinking goes to message.reasoning_content, # matching the retired v3 service so the fleet's reasoning clients work) -# - batch 4096 / ubatch 4096 (larger than the runner's 2048/512; matches v3) # - --no-mmap (local choice; benchmarks show no perf difference versus # mmap, kept per operator preference) # - chat_template_kwargs preserve_thinking=true (preserve thinking tokens