dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

bench: add prod2048 config (runner batch sizes on chat profile); docs: ubatch 4096 results

Browse Source
This commit is contained in:
Frank
2026-08-22 03:17:41 -07:00
parent bd9d75af3f
commit 3eb29de6ca
3 changed files with 35 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+6 -2
View File
@@ -137,7 +137,11 @@ The runtime routes qualified prompt and verification shapes through the
RDNA 3.5 native IU4 matrix instruction. The build uses the release script
against the private TheRock ROCm 7.15 SDK kept in the retired ling3-ciru
venv. The model files (GGUF + FFN/GDN/GDN-Output .pfs, 26.05 GiB) live in
`/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge/`.
`/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge/`. The production profile
adds batch 4096 / ubatch 4096, `--no-mmap`, and reasoning on (deepseek
format, preserve_thinking) on top of the pinned runner. Ubatch 4096 slows
prefill ~25% versus ubatch 2048 with no decode change; see
docs/kairic-edge.md before changing the batch profile.
**Release gap — CK `config.h`:** the pinned CK commit ships only
`config.h.in`; the release script and guide never generate
@@ -210,7 +214,7 @@ graph TD
- `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw Kairic loads the 26.05 GiB payload in a few minutes (page-cache warm). Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`).
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw Kairic loads the 26.05 GiB payload in ~12 s with a warm page cache, a few minutes cold. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`).
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md).
docs/kairic-edge.md
+27 -4
View File
@@ -211,7 +211,8 @@ See [[deployment]] for the update process.
|---------|-------|--------|
| Bind | `0.0.0.0:52625` | Local (runner default `127.0.0.1:8080`) |
| Context | 262144, 1 slot | Pinned runner |
| Batch / ubatch | 4096 / 2048 | Local (runner: 2048 / 512) |
| Batch / ubatch | 4096 / 4096 | Local (runner: 2048 / 512; v3: 4096 / 4096) |
| Model loading | `--no-mmap` | Local (benchmarks: no perf difference) |
| Threads | 16 target / 32 batch | Pinned runner |
| KV | f16 target and draft | Pinned runner |
| Prompt cache | `--cache-ram 8192 --cache-prompt --cache-idle-slots -ctxcp 32` | Pinned runner |
@@ -227,8 +228,12 @@ Local deviations and their reasons:
- **Reasoning on + deepseek**: thinking goes to `message.reasoning_content`,
matching the retired v3 service. `preserve_thinking=true` keeps thinking
tokens in the history across turns.
- **Batch 4096 / ubatch 2048**: larger than the runner; unified memory has
headroom. Prefill and decode match the runner profile in benchmarks.
- **Batch 4096 / ubatch 4096**: larger than the runner; matches the
retired v3 profile. Unified memory has headroom. Ubatch 4096 slows
prefill (~25% versus ubatch 2048, ~50% versus the runner's ubatch 512)
and does not change decode.
- **`--no-mmap`**: benchmarks show no performance difference versus mmap.
Kept per operator preference (2026-08-22).
- **Target greedy argmax fast path off**: the pinned runner sets
`LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1`. That fast path fails closed on
any non-greedy request (400 error). The model card's own chat
@@ -255,7 +260,8 @@ Findings:
- Prefill runs 1.5-2.3x faster than v3 at every depth.
- Prose decode runs 15-26% slower than v3 at every depth.
- `--no-mmap` makes no difference (19.89 vs 19.80 t/s decode).
- `--no-mmap` makes no difference (19.89 vs 19.80 t/s decode). The
service now runs `--no-mmap` per operator preference.
- `-c 262144` versus `-c 8192` makes no difference to decode.
- Live service at ~116k context: 13.5-14.9 t/s, consistent with the 64k
bench. Long conversations run in the deep-context regime.
@@ -266,6 +272,23 @@ Findings:
represent prose chat. The card's natural-prose slice measured 34.88
t/s on the certified system.
### ubatch 4096 + --no-mmap profile (2026-08-22)
The production profile moved from batch 4096 / ubatch 2048 with mmap to
batch 4096 / ubatch 4096 with `--no-mmap` (matching the v3 profile). Same
harness, same prompts.
| Context | ub2048 decode (cold/warm) | ub4096 decode (cold/warm) | ub2048 prefill cold | ub4096 prefill cold |
|---------|---------------------------|---------------------------|---------------------|---------------------|
| ~8k | 19.79 / 19.36 | 22.65 / 19.78 | 348 | 266 |
| ~16k | 19.83 / 18.75 | 18.13 / 22.48 | 370 | 258 |
| ~64k | 14.61 / - | 14.11 / 13.80 | 232 | 203 |
Ubatch 4096 does not change decode at any depth. It slows prefill by
~25% versus ubatch 2048 and ~50% versus the runner's ubatch 512
(506 t/s at 8k). The IU4 lane accelerates 96-512-row prompt and
verification shapes; large ubatches do not use those shapes.
## Rollback to v3
The v3 payload and binary stay on the machine for rollback.
scripts/bench-kairic-ab.sh
+2
View File
@@ -9,6 +9,7 @@
# nofast runner + fast path OFF
# prod production profile (b4096/ub4096, --no-mmap, reasoning on/deepseek,
# fast path OFF, preserve_thinking)
# prod2048 prod with the pinned runner's batch sizes (b2048/ub512)
# prodnommap (obsolete: identical to prod since ubatch 4096 + --no-mmap
# moved into the production profile)
#
@@ -63,6 +64,7 @@ case "$CFG" in
nommap) export LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1; B=2048; UB=512; REA=off; RFMT=none; EXTRA=""; MMAP="--no-mmap" ;;
nofast) export LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0; B=2048; UB=512; REA=off; RFMT=none; EXTRA="" ;;
prod) export LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0; B=4096; UB=4096; REA=on; RFMT=deepseek; EXTRA="--chat-template-kwargs {\"preserve_thinking\":\"true\"}"; MMAP="--no-mmap" ;;
prod2048) export LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0; B=2048; UB=512; REA=on; RFMT=deepseek; EXTRA="--chat-template-kwargs {\"preserve_thinking\":\"true\"}"; MMAP="--no-mmap" ;;
prodnommap) export LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0; B=4096; UB=2048; REA=on; RFMT=deepseek; EXTRA="--chat-template-kwargs {\"preserve_thinking\":\"true\"}"; MMAP="--no-mmap" ;;
esac