dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Kairic Edge reference page (build runbook, config.h gap, byte-identical rebuild, perf data); mark v3 doc retired; update topology

Browse Source
This commit is contained in:
Frank committed 2026-08-22 02:51:39 -07:00
1 parent 40d50fed06
commit 4360551938
4 files changed
+356 -40
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+42 -30
View File
@@ -3,20 +3,24 @@
## Overview
This project manages deployment scripts for the LLM inference fleet on two
local machines. Topology (2026-08-21):
local machines. Topology (2026-08-22):
| Target | Host | Runtime | Service | Model | Port |
|--------|------|---------|---------|-------|------|
| Dankserver | dank@dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy, PromptForge DualView, 262K context / 1 slot | 52625 |
| Dankclaw | dank@dankclaw | Kairic Edge CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-IU4-Kairic-Edge, native IU4 lane + MTP4 + ngram-mod, 262K context / 1 slot | 52625 |
The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw; the
CIRU ROCmFPX llama.cpp runtime runs on the Strix Halo iGPU (gfx1151). On
2026-08-21 a DFlash2 Vulkan deployment briefly replaced CIRU, then was
reverted the same day for speed; the DFlash2 files live in `.AGENTTRASH/`.
See docs/ciru-activefpx-promptforge.md. The old scripts live in
`old/scripts/` for reference; they describe the retired PP topology.
NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw;
the CIRU ROCmFPX runtime then served the v3 ActiveFPX PromptForge model on
the Strix Halo iGPU (gfx1151). On 2026-08-22 the service swapped to the
Kairic Edge IU4 runtime (pinned tag `kairic-edge-qwen38-27b-v1`); v3
files remain on the machine for rollback. On 2026-08-21 a DFlash2 Vulkan
deployment briefly replaced CIRU, then was reverted the same day for
speed; the DFlash2 files live in `.AGENTTRASH/`. See
docs/kairic-edge.md and docs/ciru-activefpx-promptforge.md. The old scripts
live in `old/scripts/` for reference; they describe the retired PP
topology.
**Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the
Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its
@@ -124,30 +128,38 @@ back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s
vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README
for the base recipe.
### CIRU ROCmFPX llama.cpp (Dankclaw)
### Kairic Edge llama.cpp (Dankclaw)
The CIRU llama-server serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge`
from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
The runtime is the ciru-ai/ROCmFPX tag `qwen3.8-activefpx-promptforge-v2.3`
(commit `103af8b41`) **plus the v3 output-K8 runtime patch** (commit
`af0eac6` on Dankclaw) built against the private TheRock ROCm 7.15 SDK kept
in the retired ling3-ciru venv
(`.venv/lib/python3.12/site-packages/_rocm_sdk_*`). The model card requires
ROCm 7.15 for the advertised Strix Halo performance; the system ROCm 7.2.4
selects slower kernels. The serve script exports the 7.15 SDK libraries
first in `LD_LIBRARY_PATH`.
The Kairic Edge llama-server serves `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge`
from `/home/dank/ciru-qwen38/ROCmFPX-kairic/build-kairic/bin/llama-server`
(version 79, commit `e97b324`, pinned tag `kairic-edge-qwen38-27b-v1`).
The runtime routes qualified prompt and verification shapes through the
RDNA 3.5 native IU4 matrix instruction. The build uses the release script
against the private TheRock ROCm 7.15 SDK kept in the retired ling3-ciru
venv. The model files (GGUF + FFN/GDN/GDN-Output .pfs, 26.05 GiB) live in
`/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge/`.
The v3 model files (GGUF + FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj,
~36.7 GiB) live in the v3 staging directory
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`; the v2 artifacts remain
in `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/` for rollback.
The full build runbook is in docs/ciru-activefpx-promptforge.md. The retired vLLM venvs
(`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to
`/home/dank/.AGENTTRASH/` on 2026-08-18.
**Release gap — CK `config.h`:** the pinned CK commit ships only
`config.h.in`; the release script and guide never generate
`third_party/composable_kernel/include/ck/config.h`. Generate it via a CK
CMake configure and copy it in before building (full runbook in
docs/kairic-edge.md). The file is untracked in CK git; a `git clean`
removes it. Local rebuilds are byte-identical (`d7625a0c...`).
**Env var naming rule still applies:** the HIP runtime reads `TEMP`, `HOST`,
and `GPU_MAX_HEAP_SIZE`. The serve script passes `--host`/`--port`/`--temp`
as CLI flags only.
**Target greedy argmax fast path:** the pinned runner sets
`LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1`; production sets it to 0 because
the fast path rejects non-greedy requests (400) and the card's own chat
sampling is non-greedy. Do not flip it back without re-testing chat.
**Rollback:** the v3 model files remain in
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/` and the v3 binary in
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
The v3 runbook is docs/ciru-activefpx-promptforge.md; the rollback
procedure is in docs/kairic-edge.md.
**Env var naming rule still applies:** the HIP runtime reads `TEMP`,
`HOST`, and `GPU_MAX_HEAP_SIZE`. The serve script passes `--host`/`--port`
as CLI flags only; bind and port come from `BIND_ADDR`/`LISTEN_PORT`.
### DFlash2 llama.cpp (Dankclaw, retired 2026-08-21)
@@ -198,7 +210,7 @@ graph TD
- `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw CIRU loads the v3 ~36.7 GiB payload in ~5-10 minutes. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`).
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw Kairic loads the 26.05 GiB payload in a few minutes (page-cache warm). Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`).
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md).