large: batch profile flipped to the runner's qualified 2048/512 (fastest measured prefill, no decode change)
This commit is contained in:
Frank
@@ -138,10 +138,11 @@ RDNA 3.5 native IU4 matrix instruction. The build uses the release script
|
||||
against the private TheRock ROCm 7.15 SDK kept in the retired ling3-ciru
|
||||
venv. The model files (GGUF + FFN/GDN/GDN-Output .pfs, 26.05 GiB) live in
|
||||
`/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge/`. The production profile
|
||||
adds batch 4096 / ubatch 4096, `--no-mmap`, and reasoning on (deepseek
|
||||
format, preserve_thinking) on top of the pinned runner. Ubatch 4096 slows
|
||||
prefill ~25% versus ubatch 2048 with no decode change; see
|
||||
docs/kairic-edge.md before changing the batch profile.
|
||||
adds `--no-mmap` and reasoning on (deepseek format, preserve_thinking) on
|
||||
top of the pinned runner; batch 2048 / ubatch 512 are the runner's
|
||||
qualified sizes. Benchmarks: ubatch 512 prefill is 60-90% faster than
|
||||
ubatch 4096 with no decode change; see docs/kairic-edge.md before
|
||||
changing the batch profile.
|
||||
|
||||
**Release gap — CK `config.h`:** the pinned CK commit ships only
|
||||
`config.h.in`; the release script and guide never generate
|
||||
|
||||
+15
-18
@@ -211,7 +211,7 @@ See [[deployment]] for the update process.
|
||||
|---------|-------|--------|
|
||||
| Bind | `0.0.0.0:52625` | Local (runner default `127.0.0.1:8080`) |
|
||||
| Context | 262144, 1 slot | Pinned runner |
|
||||
| Batch / ubatch | 4096 / 4096 | Local (runner: 2048 / 512; v3: 4096 / 4096) |
|
||||
| Batch / ubatch | 2048 / 512 | Pinned runner (fastest measured prefill) |
|
||||
| Model loading | `--no-mmap` | Local (benchmarks: no perf difference) |
|
||||
| Threads | 16 target / 32 batch | Pinned runner |
|
||||
| KV | f16 target and draft | Pinned runner |
|
||||
@@ -228,10 +228,6 @@ Local deviations and their reasons:
|
||||
- **Reasoning on + deepseek**: thinking goes to `message.reasoning_content`,
|
||||
matching the retired v3 service. `preserve_thinking=true` keeps thinking
|
||||
tokens in the history across turns.
|
||||
- **Batch 4096 / ubatch 4096**: larger than the runner; matches the
|
||||
retired v3 profile. Unified memory has headroom. Ubatch 4096 slows
|
||||
prefill (~25% versus ubatch 2048, ~50% versus the runner's ubatch 512)
|
||||
and does not change decode.
|
||||
- **`--no-mmap`**: benchmarks show no performance difference versus mmap.
|
||||
Kept per operator preference (2026-08-22).
|
||||
- **Target greedy argmax fast path off**: the pinned runner sets
|
||||
@@ -272,22 +268,23 @@ Findings:
|
||||
represent prose chat. The card's natural-prose slice measured 34.88
|
||||
t/s on the certified system.
|
||||
|
||||
### ubatch 4096 + --no-mmap profile (2026-08-22)
|
||||
### Batch profile comparison (2026-08-22)
|
||||
|
||||
The production profile moved from batch 4096 / ubatch 2048 with mmap to
|
||||
batch 4096 / ubatch 4096 with `--no-mmap` (matching the v3 profile). Same
|
||||
harness, same prompts.
|
||||
Three batch profiles ran on the identical chat profile (fast path off,
|
||||
reasoning on, `--no-mmap`; the ub2048 row was the earlier mmap run, and
|
||||
mmap makes no difference):
|
||||
|
||||
| Context | ub2048 decode (cold/warm) | ub4096 decode (cold/warm) | ub2048 prefill cold | ub4096 prefill cold |
|
||||
|---------|---------------------------|---------------------------|---------------------|---------------------|
|
||||
| ~8k | 19.79 / 19.36 | 22.65 / 19.78 | 348 | 266 |
|
||||
| ~16k | 19.83 / 18.75 | 18.13 / 22.48 | 370 | 258 |
|
||||
| ~64k | 14.61 / - | 14.11 / 13.80 | 232 | 203 |
|
||||
| Context | b2048/ub512 decode (cold/warm) | b4096/ub2048 decode | b4096/ub4096 decode | ub512 prefill cold | ub4096 prefill cold |
|
||||
|---------|--------------------------------|---------------------|---------------------|--------------------|---------------------|
|
||||
| ~8k | 19.6 / 18.7 | 19.8 / 19.4 | 22.7 / 19.8 | 524 | 266 |
|
||||
| ~16k | 21.8 / 21.5 | 19.8 / 18.8 | 18.1 / 22.5 | 485 | 258 |
|
||||
| ~64k | 13.7 / 15.7 | 14.6 / - | 14.1 / 13.8 | 323 | 203 |
|
||||
|
||||
Ubatch 4096 does not change decode at any depth. It slows prefill by
|
||||
~25% versus ubatch 2048 and ~50% versus the runner's ubatch 512
|
||||
(506 t/s at 8k). The IU4 lane accelerates 96-512-row prompt and
|
||||
verification shapes; large ubatches do not use those shapes.
|
||||
Ubatch is a prefill knob. Ubatch 512 gives the fastest prefill at every
|
||||
depth, 60-90% above ubatch 4096. Decode is identical across all three
|
||||
profiles within run-to-run noise. The IU4 lane accelerates 96-512-row
|
||||
prompt and verification shapes; ubatch 512 matches those shapes, large
|
||||
ubatches do not. Production runs the runner's qualified 2048/512.
|
||||
|
||||
## Rollback to v3
|
||||
|
||||
|
||||
@@ -50,11 +50,12 @@ LISTEN_PORT=52625
|
||||
# while the prompt cache is enabled.
|
||||
CTX_SIZE=262144
|
||||
PARALLEL=1
|
||||
# Local override: the pinned runner uses 2048/512; production chat uses a
|
||||
# larger 4096/4096 batch (unified memory has headroom; matches the retired
|
||||
# v3 profile).
|
||||
BATCH_SIZE=4096
|
||||
UBATCH_SIZE=4096
|
||||
# Batch profile: the pinned runner's qualified sizes (2048/512). Benchmarks
|
||||
# on 2026-08-22: ubatch 512 gives the fastest prefill at every depth
|
||||
# (524/485/323 t/s at 8k/16k/64k vs 266/258/203 for ubatch 4096) with no
|
||||
# decode difference. Do not raise ubatch without re-benching prefill.
|
||||
BATCH_SIZE=2048
|
||||
UBATCH_SIZE=512
|
||||
TARGET_THREADS=16
|
||||
BATCH_THREADS=32
|
||||
CACHE_TYPE_K=f16
|
||||
|
||||
@@ -16,9 +16,10 @@
|
||||
# differences: --host/--port come from BIND_ADDR/LISTEN_PORT (safe names;
|
||||
# the ROCm HIP runtime reads a bare HOST env var, see the 2026-08-13 TEMP
|
||||
# incident), LD_LIBRARY_PATH uses this machine's split 7.15 SDK layout,
|
||||
# the profile runs --no-mmap, batch 4096 / ubatch 4096, reasoning on with
|
||||
# deepseek format and preserve_thinking, and the target greedy argmax fast
|
||||
# path is disabled for non-greedy chat sampling.
|
||||
# the profile runs --no-mmap, reasoning on with deepseek format and
|
||||
# preserve_thinking, and the target greedy argmax fast path is disabled
|
||||
# for non-greedy chat sampling. Batch 2048 / ubatch 512 are the runner's
|
||||
# qualified sizes.
|
||||
#
|
||||
# Tunables come from frank-default-large.env in this directory unless
|
||||
# FRANK_LARGE_ENV_FILE overrides that path.
|
||||
@@ -49,8 +50,8 @@ fi
|
||||
: "${LISTEN_PORT:=52625}"
|
||||
: "${CTX_SIZE:=262144}"
|
||||
: "${PARALLEL:=1}"
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
: "${UBATCH_SIZE:=4096}"
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
: "${TARGET_THREADS:=16}"
|
||||
: "${BATCH_THREADS:=32}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
@@ -125,10 +126,11 @@ export PROMPTFORGE_ENABLE_IU4_DECODE_M2_M5=0
|
||||
export PROMPTFORGE_OUTPUT_K8_STRICT_GREEDY=0
|
||||
|
||||
# Pinned runner flags (qualified 262K one-slot profile, native MTP depth
|
||||
# 4, deterministic sampler). Local overrides for production chat:
|
||||
# 4, deterministic sampler; batch 2048 / ubatch 512 are the runner's
|
||||
# qualified sizes and give the fastest measured prefill). Local overrides
|
||||
# for production chat:
|
||||
# - reasoning on + deepseek format (thinking goes to message.reasoning_content,
|
||||
# matching the retired v3 service so the fleet's reasoning clients work)
|
||||
# - batch 4096 / ubatch 4096 (larger than the runner's 2048/512; matches v3)
|
||||
# - --no-mmap (local choice; benchmarks show no perf difference versus
|
||||
# mmap, kept per operator preference)
|
||||
# - chat_template_kwargs preserve_thinking=true (preserve thinking tokens
|
||||
|
||||
Reference in New Issue
Block a user