dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Revert frank-default-large to CIRU runtime (DFlash2 too slow)

Browse Source 
Restore the CIRU ROCmFPX setup that DFlash2 replaced:

- Restore jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2 runtime
  (native depth-4 MTP, PromptForge DualView, ROCm 7.15)
- Archive the DFlash2 Vulkan deployment files in .AGENTTRASH
- DFlash2 was retired the same day it shipped (2026-08-21) for speed
- Revert docs (deployment.md, index.md, AGENTS.md); mark dflash2 page archived
- Restored scripts are byte-identical to the pre-swap CIRU versions
This commit is contained in:
Frank committed 2026-08-21 10:15:23 -07:00
1 parent fd0d9dd3cb
commit 583efd8cf2
11 files changed
+316 -302
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+30 -21
View File
@@ -8,16 +8,15 @@ local machines. Topology (2026-08-21):
| Target | Host | Runtime | Service | Model | Port |
|--------|------|---------|---------|-------|------|
| Dankserver | dank@dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
| Dankclaw | dank@dankclaw | llama.cpp PR #27342 DFlash2 (Vulkan RADV) | frank-default-large | unsloth Qwen3.8-27B-UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2-Q4_K_M drafter, 262K context / 1 slot | 52625 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView, 262K context / 1 slot | 52625 |
The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
NPU endpoint. The 2026-08-18 swap retired the vLLM service and the CIRU
ROCmFPX runtime on Dankclaw. On 2026-08-21 `frank-default-large` moved to
upstream llama.cpp PR #27342 with DFlash2 speculative decoding on the
Vulkan backend (`Vulkan0`, RADV gfx1151). The retired CIRU AI and IQ4_XS
files live in `.AGENTTRASH/`. See docs/dflash2-large-vulkan.md. The old
scripts live in `old/scripts/` for reference; they describe the retired PP
topology.
NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw; the
CIRU ROCmFPX llama.cpp runtime runs on the Strix Halo iGPU (gfx1151). On
2026-08-21 a DFlash2 Vulkan deployment briefly replaced CIRU, then was
reverted the same day for speed; the DFlash2 files live in `.AGENTTRASH/`.
See docs/ciru-activefpx-promptforge.md. The old scripts live in
`old/scripts/` for reference; they describe the retired PP topology.
**Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the
Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its
@@ -63,7 +62,7 @@ Use `scripts/deploy.sh` to deploy committed changes to a target. It pushes `main
```
FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver # llama.cpp FastMTP, port 7770
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw # DFlash2 llama.cpp, port 52625
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw # CIRU llama.cpp, port 52625
./scripts/deploy.sh dankserver --dry-run # preview without changes
```
@@ -125,24 +124,34 @@ back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s
vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README
for the base recipe.
### DFlash2 llama.cpp (Dankclaw, Vulkan)
### CIRU ROCmFPX llama.cpp (Dankclaw)
The upstream llama.cpp PR #27342 llama-server serves
`unsloth/Qwen3.8-27B-UD-Q8_K_XL.gguf` with a z-lab DFlash2 drafter
(`Qwen3.8-27B-DFlash2-Q4_K_M.gguf`) on the Vulkan backend. The binary lives
at `/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server`. DFlash2
needs `TOP_K` + `ARGMAX`, which HIP does not implement, so the stack runs on
`Vulkan0` (RADV, gfx1151). See docs/dflash2-large-vulkan.md.
The CIRU llama-server serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge`
from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
The runtime is the ciru-ai/ROCmFPX tag `qwen3.8-activefpx-promptforge-v2.3`
(commit `103af8b41`) built against the private TheRock ROCm 7.15 SDK kept
in the retired ling3-ciru venv
(`.venv/lib/python3.12/site-packages/_rocm_sdk_*`). The model card requires
ROCm 7.15 for the advertised Strix Halo performance; the system ROCm 7.2.4
selects slower kernels. The serve script exports the 7.15 SDK libraries
first in `LD_LIBRARY_PATH`.
The target and drafter live in `/home/dank/models/`. The retired CIRU
runtime (ROCmFPX v2.3, ROCm 7.15 SDK in the ling3-ciru venv, FFN/GDN
companions) is archived in `.AGENTTRASH/` and documented in
circu-activefpx-promptforge.md.
The model files (v2 GGUF + FFN.pfs + GDN.pfs, ~34.9 GiB) live in
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/`. The full build runbook is
in docs/ciru-activefpx-promptforge.md. The retired vLLM venvs
(`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to
`/home/dank/.AGENTTRASH/` on 2026-08-18.
**Env var naming rule still applies:** the HIP runtime reads `TEMP`, `HOST`,
and `GPU_MAX_HEAP_SIZE`. The serve script passes `--host`/`--port`/`--temp`
as CLI flags only.
### DFlash2 llama.cpp (Dankclaw, retired 2026-08-21)
A DFlash2 Vulkan deployment briefly replaced CIRU on 2026-08-21, then was
reverted the same day for speed. Its serve script, env, and the build worktree
are archived in `.AGENTTRASH/` and documented in docs/dflash2-large-vulkan.md.
### Env var naming rule — HIP runtime collision
The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`,
@@ -186,7 +195,7 @@ graph TD
- `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw DFlash2 loads the 31.4 GB UD-Q8_K_XL + 2.0 GB drafter in ~1 minute with a warm OS page cache and a few minutes cold. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`).
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw CIRU loads the 34.9 GiB ActiveFPX payload in ~5-10 minutes. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`).
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md).