bench: add prod2048 config (runner batch sizes on chat profile); docs: ubatch 4096 results
This commit is contained in:
1 parent
bd9d75af3f
commit
3eb29de6ca
3 files changed
+35
-6
No files matched your search
@@ -137,7 +137,11 @@ The runtime routes qualified prompt and verification shapes through the
|
||||
RDNA 3.5 native IU4 matrix instruction. The build uses the release script
|
||||
against the private TheRock ROCm 7.15 SDK kept in the retired ling3-ciru
|
||||
venv. The model files (GGUF + FFN/GDN/GDN-Output .pfs, 26.05 GiB) live in
|
||||
`/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge/`.
|
||||
`/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge/`. The production profile
|
||||
adds batch 4096 / ubatch 4096, `--no-mmap`, and reasoning on (deepseek
|
||||
format, preserve_thinking) on top of the pinned runner. Ubatch 4096 slows
|
||||
prefill ~25% versus ubatch 2048 with no decode change; see
|
||||
docs/kairic-edge.md before changing the batch profile.
|
||||
|
||||
**Release gap — CK `config.h`:** the pinned CK commit ships only
|
||||
`config.h.in`; the release script and guide never generate
|
||||
@@ -210,7 +214,7 @@ graph TD
|
||||
|
||||
- `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
|
||||
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
|
||||
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw Kairic loads the 26.05 GiB payload in a few minutes (page-cache warm). Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`).
|
||||
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw Kairic loads the 26.05 GiB payload in ~12 s with a warm page cache, a few minutes cold. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`).
|
||||
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
|
||||
- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md).
|
||||
|
||||
|
||||
Reference in new issue
Block a user