dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

large: batch profile flipped to the runner's qualified 2048/512 (fastest measured prefill, no decode change)

Browse Source
This commit is contained in:
Frank
2026-08-22 03:30:58 -07:00
parent 3eb29de6ca
commit 8b3cd2ff79
4 changed files with 35 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/kairic-edge.md
+15 -18
View File
@@ -211,7 +211,7 @@ See [[deployment]] for the update process.
|---------|-------|--------|
| Bind | `0.0.0.0:52625` | Local (runner default `127.0.0.1:8080`) |
| Context | 262144, 1 slot | Pinned runner |
| Batch / ubatch | 4096 / 4096 | Local (runner: 2048 / 512; v3: 4096 / 4096) |
| Batch / ubatch | 2048 / 512 | Pinned runner (fastest measured prefill) |
| Model loading | `--no-mmap` | Local (benchmarks: no perf difference) |
| Threads | 16 target / 32 batch | Pinned runner |
| KV | f16 target and draft | Pinned runner |
@@ -228,10 +228,6 @@ Local deviations and their reasons:
- **Reasoning on + deepseek**: thinking goes to `message.reasoning_content`,
matching the retired v3 service. `preserve_thinking=true` keeps thinking
tokens in the history across turns.
- **Batch 4096 / ubatch 4096**: larger than the runner; matches the
retired v3 profile. Unified memory has headroom. Ubatch 4096 slows
prefill (~25% versus ubatch 2048, ~50% versus the runner's ubatch 512)
and does not change decode.
- **`--no-mmap`**: benchmarks show no performance difference versus mmap.
Kept per operator preference (2026-08-22).
- **Target greedy argmax fast path off**: the pinned runner sets
@@ -272,22 +268,23 @@ Findings:
represent prose chat. The card's natural-prose slice measured 34.88
t/s on the certified system.
### ubatch 4096 + --no-mmap profile (2026-08-22)
### Batch profile comparison (2026-08-22)
The production profile moved from batch 4096 / ubatch 2048 with mmap to
batch 4096 / ubatch 4096 with `--no-mmap` (matching the v3 profile). Same
harness, same prompts.
Three batch profiles ran on the identical chat profile (fast path off,
reasoning on, `--no-mmap`; the ub2048 row was the earlier mmap run, and
mmap makes no difference):
| Context | ub2048 decode (cold/warm) | ub4096 decode (cold/warm) | ub2048 prefill cold | ub4096 prefill cold |
|---------|---------------------------|---------------------------|---------------------|---------------------|
| ~8k | 19.79 / 19.36 | 22.65 / 19.78 | 348 | 266 |
| ~16k | 19.83 / 18.75 | 18.13 / 22.48 | 370 | 258 |
| ~64k | 14.61 / - | 14.11 / 13.80 | 232 | 203 |
| Context | b2048/ub512 decode (cold/warm) | b4096/ub2048 decode | b4096/ub4096 decode | ub512 prefill cold | ub4096 prefill cold |
|---------|--------------------------------|---------------------|---------------------|--------------------|---------------------|
| ~8k | 19.6 / 18.7 | 19.8 / 19.4 | 22.7 / 19.8 | 524 | 266 |
| ~16k | 21.8 / 21.5 | 19.8 / 18.8 | 18.1 / 22.5 | 485 | 258 |
| ~64k | 13.7 / 15.7 | 14.6 / - | 14.1 / 13.8 | 323 | 203 |
Ubatch 4096 does not change decode at any depth. It slows prefill by
~25% versus ubatch 2048 and ~50% versus the runner's ubatch 512
(506 t/s at 8k). The IU4 lane accelerates 96-512-row prompt and
verification shapes; large ubatches do not use those shapes.
Ubatch is a prefill knob. Ubatch 512 gives the fastest prefill at every
depth, 60-90% above ubatch 4096. Decode is identical across all three
profiles within run-to-run noise. The IU4 lane accelerates 96-512-row
prompt and verification shapes; ubatch 512 matches those shapes, large
ubatches do not. Production runs the runner's qualified 2048/512.
## Rollback to v3