Move frank-default-large from :52625 to :7770 (dankclaw)
Operator request (2026-08-22): the primary inference server (frank-default-large, the 27B CIRU model on Dankclaw) moves to the standard llama.cpp port 7770. Both LLM services now listen on :7770, distinguished by host (dankserver vs dankclaw). Functional changes: - scripts/frank-default-large.env: LISTEN_PORT 52625 -> 7770 - scripts/serve-frank-large.sh: default LISTEN_PORT 52625 -> 7770 - scripts/deploy.sh: LLM_HEALTH[dankclaw] -> :7770 - open-webui quadlet: OPENAI_API_BASE_URL -> 192.168.0.143:7770 (re-pointed so chat keeps working; data dir preserved) - scripts/bench-fast-greedy.sh: PORT 52625 -> 7770 Tests updated to the new topology (both LLM endpoints on :7770 by host; open-webui -> :7770). Docs + pi KB updated; dated historical runbooks keep their point-in-time ports. models.json (pi client) re-pointed to :7770 separately, effective on the next client restart.
This commit is contained in:
Frank
@@ -1,7 +1,7 @@
|
||||
# Dankclaw Stack
|
||||
|
||||
Dankclaw (192.168.0.143) runs the AI, search, and development stack. It
|
||||
hosts the LLM inference endpoint (`frank-default-large`, `:52625`, CIRU
|
||||
hosts the LLM inference endpoint (`frank-default-large`, `:7770`, CIRU
|
||||
ROCmFPX llama.cpp), the agent tools, the privacy search, and the
|
||||
observability stack. The llama.cpp endpoint (`frank-default-small`,
|
||||
`:7770`) runs on Dankserver (192.168.0.235).
|
||||
@@ -17,7 +17,7 @@ graph LR
|
||||
S --> V[Gluetun VPN]
|
||||
O --> L[llama.cpp dankserver :7770]
|
||||
O --> H[Hermes]
|
||||
O --> E[CIRU llama.cpp :52625]
|
||||
O --> E[CIRU llama.cpp :7770]
|
||||
```
|
||||
|
||||
## Services
|
||||
@@ -32,7 +32,7 @@ graph LR
|
||||
| Grafana | grafana.dankclaw.local | 3002 | [[Grafana]] |
|
||||
| Prometheus | prometheus.dankclaw.local | 9090 | [[Prometheus]] |
|
||||
| Homepage | dankclaw.local | 80/443 | — |
|
||||
| frank-default-large (host service, Dankclaw) | — | 52625 | [[deployment]] |
|
||||
| frank-default-large (host service, Dankclaw) | — | 7770 | [[deployment]] |
|
||||
| frank-default-small (host service, Dankserver) | — | 7770 | [[deployment]] |
|
||||
|
||||
The last two rows are bare-metal systemd services, not containers. This
|
||||
|
||||
@@ -11,19 +11,19 @@ OpenWebUI is a self-hosted chat interface for LLMs. It connects to OpenAI-compat
|
||||
| Image | `ghcr.io/open-webui/open-webui:main` (CPU build; Dankclaw is CPU-only) |
|
||||
| Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
|
||||
| Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
|
||||
| Default backend | vLLM on Dankclaw `:52625` = `frank-default-large` (via host LAN IP `192.168.0.143:52625`) |
|
||||
| Default backend | CIRU llama.cpp on Dankclaw `:7770` = `frank-default-large` (via host LAN IP `192.168.0.143:7770`) |
|
||||
| Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) |
|
||||
| Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` |
|
||||
| Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` |
|
||||
|
||||
## The LLM backends
|
||||
|
||||
The fleet runs two OpenAI-compatible endpoints. This mapping is the **live** state (verified 2026-08-19); it is the reverse of the 2026-08-17 note, which is now out of date.
|
||||
The fleet runs two OpenAI-compatible endpoints. This mapping is the **live** state (verified 2026-08-22). Both services listen on `:7770`, distinguished by host. On 2026-08-22 `frank-default-large` moved from `:52625` to `:7770`.
|
||||
|
||||
| Endpoint | Model name | Host | Capability |
|
||||
|----------|-----------|------|-----------|
|
||||
| `:52625` (vLLM) | `frank-default-large` | Dankclaw | completion |
|
||||
| `:7770` (llama.cpp FastMTP) | `frank-default-small` | Dankserver | completion, multimodal |
|
||||
| `:7770` (CIRU llama.cpp) | `frank-default-large` | Dankclaw | completion |
|
||||
| `:7770` (llama.cpp) | `frank-default-small` | Dankserver | completion, multimodal |
|
||||
|
||||
The default connection points at the local vLLM (`frank-default-large`), so chat stays on-host. To add the other backend, open **Admin → Connections** in the UI and point an OpenAI connection at `http://192.168.0.235:7770/v1` (key `none`).
|
||||
|
||||
@@ -61,7 +61,7 @@ The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start
|
||||
- Use the non-CUDA image on this CPU-only host.
|
||||
- The default admin account is created on first login; secure it immediately.
|
||||
- If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
|
||||
- The LLM is reached via the host LAN IP `192.168.0.143:52625`, not the bridge gateway: the gateway (`10.89.3.1`) lives in the rootless netns, while llama-server runs in the host netns, so the gateway refuses. `host.containers.internal` is slirp4netns-only and does not resolve on a bridge either. To reach the Dankserver backend, use the LAN address `192.168.0.235:7770`.
|
||||
- The LLM is reached via the host LAN IP `192.168.0.143:7770`, not the bridge gateway: the gateway (`10.89.3.1`) lives in the rootless netns, while llama-server runs in the host netns, so the gateway refuses. `host.containers.internal` is slirp4netns-only and does not resolve on a bridge either. To reach the Dankserver backend, use the LAN address `192.168.0.235:7770`.
|
||||
|
||||
## History
|
||||
|
||||
|
||||
@@ -93,7 +93,7 @@ cable between claw and server is required for 10G.
|
||||
|
||||
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
|
||||
- The GPU budget is the rocm-smi figure: 64 GiB VRAM reported by rocm-smi (SVM-backed). The older 96 GiB figure was the addressable SVM pool, not the budget. Trust rocm-smi and the runtime; do not use `free` or `dmidecode` for the GPU budget.
|
||||
- The active service is `frank-default-large` on port 52625: CIRU ROCmFPX llama.cpp serving jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (262144 ctx, native MTP, PromptForge DualView). See [[ciru-activefpx-promptforge]].
|
||||
- The active service is `frank-default-large` on port 7770 (moved from 52625 on 2026-08-22): CIRU ROCmFPX llama.cpp serving jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (262144 ctx, native MTP, PromptForge DualView). See [[ciru-activefpx-promptforge]].
|
||||
- The runtime requires ROCm 7.15 (model card requirement). The system ROCm is 7.2.4; the retired ling3-ciru venv bundles a self-contained TheRock ROCm 7.15 SDK that the serve script puts first in `LD_LIBRARY_PATH`.
|
||||
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
|
||||
- 1.1 TB free disk space is ample for model storage.
|
||||
|
||||
@@ -19,7 +19,7 @@ timestamp: 2026-08-22
|
||||
> decision). Commit `c60156f` restored the v3 script pair. See
|
||||
> [[kairic-edge]] for the retired experiment reference.
|
||||
|
||||
`frank-default-large` (port 52625) serves
|
||||
`frank-default-large` (port 7770, moved from 52625 on 2026-08-22) serves
|
||||
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` **v3** through the CIRU
|
||||
ROCmFPX llama.cpp runtime. This replaced the retired vLLM service that
|
||||
previously occupied the slot. The v2 deployment moved to v3 on 2026-08-21.
|
||||
@@ -150,7 +150,7 @@ against the model card before the service restart.
|
||||
|
||||
## Service
|
||||
|
||||
- Unit: `frank-default-large.service`, port 52625
|
||||
- Unit: `frank-default-large.service`, port 7770
|
||||
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-default-large.env`
|
||||
- The script exports the ROCm 7.15 SDK libraries in `LD_LIBRARY_PATH` before
|
||||
the system ROCm. `ROCM_PATH` points at the devel SDK.
|
||||
@@ -197,7 +197,7 @@ it and generation proceeds.
|
||||
|
||||
## Operational Notes
|
||||
|
||||
- Health: `curl http://127.0.0.1:52625/health`. Returns 503 while loading.
|
||||
- Health: `curl http://127.0.0.1:7770/health`. Returns 503 while loading.
|
||||
- Model load is slow (v3 five-file payload ~36.7 GiB); do not kill the
|
||||
process during warmup. The page cache makes a repeat load fast right after
|
||||
download, but a later cold start takes several minutes.
|
||||
|
||||
@@ -18,7 +18,7 @@ repository to a running service.
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 |
|
||||
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 52625 |
|
||||
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 7770 |
|
||||
|
||||
## Workflow
|
||||
|
||||
@@ -87,7 +87,7 @@ not used.
|
||||
Health endpoints:
|
||||
|
||||
- Dankserver: `http://127.0.0.1:7770/health`
|
||||
- Dankclaw: `http://127.0.0.1:52625/health`
|
||||
- Dankclaw: `http://127.0.0.1:7770/health`
|
||||
|
||||
## Manual Update
|
||||
|
||||
@@ -114,7 +114,7 @@ git pull origin main
|
||||
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart frank-default-large
|
||||
curl -sf http://127.0.0.1:52625/health
|
||||
curl -sf http://127.0.0.1:7770/health
|
||||
```
|
||||
|
||||
## Service Units
|
||||
@@ -137,7 +137,7 @@ launches through `scripts/serve-frank-small.sh`, which reads tunables from
|
||||
The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
|
||||
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v3 five-file payload: GGUF
|
||||
+ FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj) under the alias
|
||||
`frank-default-large` on port 52625. It runs
|
||||
`frank-default-large` on port 7770 (moved from 52625 on 2026-08-22). It runs
|
||||
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
|
||||
7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP
|
||||
(`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context
|
||||
|
||||
+2
-2
@@ -26,14 +26,14 @@ ngram-mod, 36 MoE experts on CPU, 700K ctx]
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
D[gfx1151 iGPU 64 GiB unified memory]
|
||||
E[CIRU ROCmFPX llama.cpp PromptForge v3
|
||||
frank-default-large, port 52625]
|
||||
frank-default-large, port 7770]
|
||||
F[jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3
|
||||
262K ctx / 1 slot, full-W8 views, Output-K8 MTP, BF16 vision]
|
||||
D --> E --> F
|
||||
end
|
||||
|
||||
I[LLM Clients / API] -->|HTTP :7770| B
|
||||
I -->|HTTP :52625| E
|
||||
I -->|HTTP :7770| E
|
||||
```
|
||||
|
||||
## Servers
|
||||
|
||||
Reference in New Issue
Block a user