dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Move frank-default-large from :52625 to :7770 (dankclaw)

Browse Source 
Operator request (2026-08-22): the primary inference server
(frank-default-large, the 27B CIRU model on Dankclaw) moves to the
standard llama.cpp port 7770. Both LLM services now listen on :7770,
distinguished by host (dankserver vs dankclaw).

Functional changes:
- scripts/frank-default-large.env: LISTEN_PORT 52625 -> 7770
- scripts/serve-frank-large.sh: default LISTEN_PORT 52625 -> 7770
- scripts/deploy.sh: LLM_HEALTH[dankclaw] -> :7770
- open-webui quadlet: OPENAI_API_BASE_URL -> 192.168.0.143:7770
  (re-pointed so chat keeps working; data dir preserved)
- scripts/bench-fast-greedy.sh: PORT 52625 -> 7770

Tests updated to the new topology (both LLM endpoints on :7770 by host;
open-webui -> :7770). Docs + pi KB updated; dated historical runbooks
keep their point-in-time ports. models.json (pi client) re-pointed to
:7770 separately, effective on the next client restart.
This commit is contained in:
Frank
2026-08-22 22:11:53 -07:00
parent e367f0f43c
commit fbfbc5c6db
15 changed files with 51 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+5 -5
View File
@@ -27,7 +27,7 @@ host.
| Target | Runtime | Service | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
| Dankclaw | CIRU ROCmFPX llama.cpp, PromptForge v3 runtime (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3 (Q8-FFNEQ-LateQ6, full-W8 views, Output-K8 MTP proxy, BF16 vision), 262K context / 1 slot | 52625 |
| Dankclaw | CIRU ROCmFPX llama.cpp, PromptForge v3 runtime (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3 (Q8-FFNEQ-LateQ6, full-W8 views, Output-K8 MTP proxy, BF16 vision), 262K context / 1 slot | 7770 |
History: the 2026-08-17 redo retired the pipeline-parallel split and the
FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
@@ -71,7 +71,7 @@ Use `scripts/deploy.sh` to deploy committed changes to a target. One script
covers both workloads per host: the podman stack and the LLM service.
```
./scripts/deploy.sh dankclaw # AI stack + frank-default-large (:52625)
./scripts/deploy.sh dankclaw # AI stack + frank-default-large (:7770)
./scripts/deploy.sh dankserver # media stack + frank-default-small (:7770)
./scripts/deploy.sh dankclaw --dry-run # preview without changes
```
@@ -147,11 +147,11 @@ the bench finishes.
1. Stop the service before any benchmark:
`sudo systemctl stop frank-default-small` (Dankserver) or
`sudo systemctl stop frank-default-large` (Dankclaw).
2. Run the bench on a scratch port (never the service ports 7770/52625).
2. Run the bench on a scratch port (never the LLM service port 7770).
3. Restart the service when the bench completes: `sudo systemctl start <svc>`.
4. Verify health before reporting:
`curl http://192.168.0.235:7770/health` or
`curl http://192.168.0.143:52625/health`.
`curl http://192.168.0.143:7770/health`.
## Pi Model Config (`~/.pi/agent/models.json`)
@@ -159,7 +159,7 @@ The Pi agent reaches the fleet through `/home/dank/.pi/agent/models.json`.
This file lives outside the repository. Each provider is one host endpoint.
Each model `id` is the served alias. The two aliases are
`frank-default-small` (Dankserver `:7770`) and `frank-default-large`
(Dankclaw `:52625`).
(Dankclaw `:7770`).
The model `id` values are stable. The file
`/home/dank/.pi/agent/settings.json` references them. Two resolution rules
README.md
+2 -2
View File
@@ -13,7 +13,7 @@ one deploy script.
| Host | User | IP | Podman stack | LLM service | LLM port |
|------|------|----|--------------|-------------|----------|
| dankserver | dank | 192.168.0.235 | media and library | frank-default-small (llama.cpp, Qwen3.6-35B-A3B) | 7770 |
| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-default-large (CIRU ROCmFPX, Qwen3.8-27B) | 52625 |
| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-default-large (CIRU ROCmFPX, Qwen3.8-27B) | 7770 |
## Active Services
@@ -30,7 +30,7 @@ one deploy script.
| Gitea | dankclaw | gitea.dankclaw.local | 3001 |
| Grafana | dankclaw | grafana.dankclaw.local | 3002 |
| Prometheus | dankclaw | prometheus.dankclaw.local | 9090 |
| frank-default-large | dankclaw | — (bare endpoint) | 52625 |
| frank-default-large | dankclaw | — (bare endpoint) | 7770 |
## Structure
dankclaw/open-webui/quadlet/open-webui.container
+3 -3
View File
@@ -5,8 +5,8 @@
#
# Chat UI for the LLM fleet. It runs on the CPU image (dankclaw is CPU-only, so
# the :cuda image would waste a large pull for a GPU it cannot use) and points
# its OpenAI connection at the live vLLM backend on this host via the bridge
# gateway (192.168.0.143:52625, model frank-default-large). It publishes 3005
# its OpenAI connection at the live llama.cpp backend on this host via the
# host LAN IP (192.168.0.143:7770, model frank-default-large). It publishes 3005
# (3000 is Gitea's HTTP port). Data persists under
# /home/dank/Containers/open-webui/data. See docs/Dankclaw/OpenWebUI.md.
#
@@ -29,7 +29,7 @@ ContainerName=open-webui
Network=open-webui.network
Image=ghcr.io/open-webui/open-webui:main
PublishPort=3005:8080
Environment=OPENAI_API_BASE_URL=http://192.168.0.143:52625
Environment=OPENAI_API_BASE_URL=http://192.168.0.143:7770
Environment=ANONYMIZED_TELEMETRY=False
Environment=DO_NOT_TRACK=True
Volume=/home/dank/Containers/open-webui/data:/app/backend/data
docs/Dankclaw/Dankclaw Stack.md
+3 -3
View File
@@ -1,7 +1,7 @@
# Dankclaw Stack
Dankclaw (192.168.0.143) runs the AI, search, and development stack. It
hosts the LLM inference endpoint (`frank-default-large`, `:52625`, CIRU
hosts the LLM inference endpoint (`frank-default-large`, `:7770`, CIRU
ROCmFPX llama.cpp), the agent tools, the privacy search, and the
observability stack. The llama.cpp endpoint (`frank-default-small`,
`:7770`) runs on Dankserver (192.168.0.235).
@@ -17,7 +17,7 @@ graph LR
S --> V[Gluetun VPN]
O --> L[llama.cpp dankserver :7770]
O --> H[Hermes]
O --> E[CIRU llama.cpp :52625]
O --> E[CIRU llama.cpp :7770]
```
## Services
@@ -32,7 +32,7 @@ graph LR
| Grafana | grafana.dankclaw.local | 3002 | [[Grafana]] |
| Prometheus | prometheus.dankclaw.local | 9090 | [[Prometheus]] |
| Homepage | dankclaw.local | 80/443 | — |
| frank-default-large (host service, Dankclaw) | — | 52625 | [[deployment]] |
| frank-default-large (host service, Dankclaw) | — | 7770 | [[deployment]] |
| frank-default-small (host service, Dankserver) | — | 7770 | [[deployment]] |
The last two rows are bare-metal systemd services, not containers. This
docs/Dankclaw/OpenWebUI.md
+5 -5
View File
@@ -11,19 +11,19 @@ OpenWebUI is a self-hosted chat interface for LLMs. It connects to OpenAI-compat
| Image | `ghcr.io/open-webui/open-webui:main` (CPU build; Dankclaw is CPU-only) |
| Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
| Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
| Default backend | vLLM on Dankclaw `:52625` = `frank-default-large` (via host LAN IP `192.168.0.143:52625`) |
| Default backend | CIRU llama.cpp on Dankclaw `:7770` = `frank-default-large` (via host LAN IP `192.168.0.143:7770`) |
| Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) |
| Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` |
| Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` |
## The LLM backends
The fleet runs two OpenAI-compatible endpoints. This mapping is the **live** state (verified 2026-08-19); it is the reverse of the 2026-08-17 note, which is now out of date.
The fleet runs two OpenAI-compatible endpoints. This mapping is the **live** state (verified 2026-08-22). Both services listen on `:7770`, distinguished by host. On 2026-08-22 `frank-default-large` moved from `:52625` to `:7770`.
| Endpoint | Model name | Host | Capability |
|----------|-----------|------|-----------|
| `:52625` (vLLM) | `frank-default-large` | Dankclaw | completion |
| `:7770` (llama.cpp FastMTP) | `frank-default-small` | Dankserver | completion, multimodal |
| `:7770` (CIRU llama.cpp) | `frank-default-large` | Dankclaw | completion |
| `:7770` (llama.cpp) | `frank-default-small` | Dankserver | completion, multimodal |
The default connection points at the local vLLM (`frank-default-large`), so chat stays on-host. To add the other backend, open **Admin → Connections** in the UI and point an OpenAI connection at `http://192.168.0.235:7770/v1` (key `none`).
@@ -61,7 +61,7 @@ The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start
- Use the non-CUDA image on this CPU-only host.
- The default admin account is created on first login; secure it immediately.
- If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
- The LLM is reached via the host LAN IP `192.168.0.143:52625`, not the bridge gateway: the gateway (`10.89.3.1`) lives in the rootless netns, while llama-server runs in the host netns, so the gateway refuses. `host.containers.internal` is slirp4netns-only and does not resolve on a bridge either. To reach the Dankserver backend, use the LAN address `192.168.0.235:7770`.
- The LLM is reached via the host LAN IP `192.168.0.143:7770`, not the bridge gateway: the gateway (`10.89.3.1`) lives in the rootless netns, while llama-server runs in the host netns, so the gateway refuses. `host.containers.internal` is slirp4netns-only and does not resolve on a bridge either. To reach the Dankserver backend, use the LAN address `192.168.0.235:7770`.
## History
docs/LLM/Dankclaw System Specs.md
+1 -1
View File
@@ -93,7 +93,7 @@ cable between claw and server is required for 10G.
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
- The GPU budget is the rocm-smi figure: 64 GiB VRAM reported by rocm-smi (SVM-backed). The older 96 GiB figure was the addressable SVM pool, not the budget. Trust rocm-smi and the runtime; do not use `free` or `dmidecode` for the GPU budget.
- The active service is `frank-default-large` on port 52625: CIRU ROCmFPX llama.cpp serving jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (262144 ctx, native MTP, PromptForge DualView). See [[ciru-activefpx-promptforge]].
- The active service is `frank-default-large` on port 7770 (moved from 52625 on 2026-08-22): CIRU ROCmFPX llama.cpp serving jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (262144 ctx, native MTP, PromptForge DualView). See [[ciru-activefpx-promptforge]].
- The runtime requires ROCm 7.15 (model card requirement). The system ROCm is 7.2.4; the retired ling3-ciru venv bundles a self-contained TheRock ROCm 7.15 SDK that the serve script puts first in `LD_LIBRARY_PATH`.
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
- 1.1 TB free disk space is ample for model storage.
docs/LLM/ciru-activefpx-promptforge.md
+3 -3
View File
@@ -19,7 +19,7 @@ timestamp: 2026-08-22
> decision). Commit `c60156f` restored the v3 script pair. See
> [[kairic-edge]] for the retired experiment reference.
`frank-default-large` (port 52625) serves
`frank-default-large` (port 7770, moved from 52625 on 2026-08-22) serves
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` **v3** through the CIRU
ROCmFPX llama.cpp runtime. This replaced the retired vLLM service that
previously occupied the slot. The v2 deployment moved to v3 on 2026-08-21.
@@ -150,7 +150,7 @@ against the model card before the service restart.
## Service
- Unit: `frank-default-large.service`, port 52625
- Unit: `frank-default-large.service`, port 7770
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-default-large.env`
- The script exports the ROCm 7.15 SDK libraries in `LD_LIBRARY_PATH` before
the system ROCm. `ROCM_PATH` points at the devel SDK.
@@ -197,7 +197,7 @@ it and generation proceeds.
## Operational Notes
- Health: `curl http://127.0.0.1:52625/health`. Returns 503 while loading.
- Health: `curl http://127.0.0.1:7770/health`. Returns 503 while loading.
- Model load is slow (v3 five-file payload ~36.7 GiB); do not kill the
process during warmup. The page cache makes a repeat load fast right after
download, but a later cold start takes several minutes.
docs/LLM/deployment.md
+4 -4
View File
@@ -18,7 +18,7 @@ repository to a running service.
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 52625 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 7770 |
## Workflow
@@ -87,7 +87,7 @@ not used.
Health endpoints:
- Dankserver: `http://127.0.0.1:7770/health`
- Dankclaw: `http://127.0.0.1:52625/health`
- Dankclaw: `http://127.0.0.1:7770/health`
## Manual Update
@@ -114,7 +114,7 @@ git pull origin main
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:52625/health
curl -sf http://127.0.0.1:7770/health
```
## Service Units
@@ -137,7 +137,7 @@ launches through `scripts/serve-frank-small.sh`, which reads tunables from
The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v3 five-file payload: GGUF
+ FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj) under the alias
`frank-default-large` on port 52625. It runs
`frank-default-large` on port 7770 (moved from 52625 on 2026-08-22). It runs
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP
(`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context
docs/LLM/index.md
+2 -2
View File
@@ -26,14 +26,14 @@ ngram-mod, 36 MoE experts on CPU, 700K ctx]
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
D[gfx1151 iGPU 64 GiB unified memory]
E[CIRU ROCmFPX llama.cpp PromptForge v3
frank-default-large, port 52625]
frank-default-large, port 7770]
F[jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3
262K ctx / 1 slot, full-W8 views, Output-K8 MTP, BF16 vision]
D --> E --> F
end
I[LLM Clients / API] -->|HTTP :7770| B
I -->|HTTP :52625| E
I -->|HTTP :7770| E
```
## Servers
scripts/bench-fast-greedy.sh
+1 -1
View File
@@ -7,7 +7,7 @@
set -euo pipefail
HOST="127.0.0.1"
PORT=52625
PORT=7770
ALIAS="frank-default-large"
# A coding prompt representative of the card's 164-task suite.
scripts/deploy.sh
+2 -2
View File
@@ -21,7 +21,7 @@
#
# Targets:
# dankclaw - AI/search/dev/observability stack + frank-default-large
# (CIRU ROCmFPX llama.cpp, Qwen3.8-27B, port 52625)
# (CIRU ROCmFPX llama.cpp, Qwen3.8-27B, port 7770)
# dankserver - media/library stack (bare containers) + frank-default-small
# (upstream llama.cpp, Qwen3.6-35B-A3B, port 7770)
#
@@ -51,7 +51,7 @@ COPY_SHARED[dankclaw]=1
# picks up the new paths at its next manual start.
LLM_SERVICES[dankclaw]="frank-default-large"
LLM_UNITS[dankclaw]="frank-default-large frank-default-npu"
LLM_HEALTH[dankclaw]="http://127.0.0.1:52625/health"
LLM_HEALTH[dankclaw]="http://127.0.0.1:7770/health"
# CIRU llama.cpp (ROCm 7.15) loads the v3 five-file payload (GGUF + FFN +
# GDN + Output-K8 .pfs + mmproj, ~36.7 GiB) + MTP draft/Output-K8 proxy and
# initializes PromptForge views; allow 20 minutes.
scripts/frank-default-large.env
+4 -2
View File
@@ -6,7 +6,9 @@
# serving jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3). v3 tensor
# allocation (FFN-equalized, late-Q6 writers, full-Q8 output) with the
# full-W8 PromptForge companions, the Output-K8 draft proxy, and the BF16
# vision projector. Alias frank-default-large, port 52625.
# vision projector. Alias frank-default-large, port 7770 (moved from 52625
# on 2026-08-22; the primary inference server sits on the standard
# llama.cpp port).
#
# This runtime replaced the retired vLLM service that previously occupied
# this slot. See docs/ciru-activefpx-promptforge.md.
@@ -41,7 +43,7 @@ ROCM_CORE=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/
ALIAS=frank-default-large
BIND_ADDR=0.0.0.0
LISTEN_PORT=52625
LISTEN_PORT=7770
# Serving: full native 262144 context (n_ctx_train), 1 slot (user request
# 2026-08-19: parallel slots degraded performance). -c is the TOTAL
scripts/serve-frank-large.sh
+1 -1
View File
@@ -41,7 +41,7 @@ fi
: "${ROCM_CORE:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_core}"
: "${ALIAS:=frank-default-large}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=52625}"
: "${LISTEN_PORT:=7770}"
: "${CTX_SIZE:=524288}"
: "${PARALLEL:=2}"
: "${BATCH_SIZE:=4096}"
tests/deployment_flow.bats
+9 -6
View File
@@ -55,17 +55,20 @@ load helpers
}
# The unified deploy.sh must manage the LLM fleet services on both hosts:
# frank-default-small (dankserver, :7770) and frank-default-large (dankclaw,
# :52625). A deploy that only restarts the podman stack leaves the LLM
# units pointing at the retired vllmScripts checkout.
# frank-default-small (dankserver) and frank-default-large (dankclaw). A
# deploy that only restarts the podman stack leaves the LLM units pointing
# at the retired vllmScripts checkout.
@test "deploy.sh restarts the LLM services on both targets" {
grep -q 'frank-default-small' "$REPO_ROOT/scripts/deploy.sh"
grep -q 'frank-default-large' "$REPO_ROOT/scripts/deploy.sh"
}
@test "deploy.sh health-checks both LLM endpoints" {
grep -qE '7770' "$REPO_ROOT/scripts/deploy.sh"
grep -qE '52625' "$REPO_ROOT/scripts/deploy.sh"
@test "deploy.sh health-checks both LLM endpoints (both on :7770, by host)" {
# 2026-08-22: frank-default-large moved from :52625 to :7770 on
# dankclaw. Both LLM services now listen on :7770, distinguished by
# host (dankserver vs dankclaw).
grep -qE '^LLM_HEALTH\[dankserver\]="http://127\.0\.0\.1:7770/health"' "$REPO_ROOT/scripts/deploy.sh"
grep -qE '^LLM_HEALTH\[dankclaw\]="http://127\.0\.0\.1:7770/health"' "$REPO_ROOT/scripts/deploy.sh"
}
@test "deploy.sh pulls on the merged checkout on both targets" {
tests/open_webui_quadlet.bats
+6 -5
View File
@@ -3,8 +3,8 @@
#
# The old open-webui was a bare container on dankserver (port 3000) pointing at a
# dead LLM backend (:9090). This unit stands one up on dankclaw on a free port
# (3005), on the CPU image (dankclaw is CPU-only), wired to the live vLLM
# backend (host.containers.internal:52625, model frank-default-large).
# (3005), on the CPU image (dankclaw is CPU-only), wired to the live
# llama.cpp backend (192.168.0.143:7770, model frank-default-large).
load helpers
@@ -58,14 +58,15 @@ QDIR="$REPO_ROOT/dankclaw/open-webui/quadlet"
|| { echo "open-webui.container must publish 3005:8080"; return 1; }
}
@test "open-webui points its LLM at the live vLLM backend (52625) via the host LAN IP" {
@test "open-webui points its LLM at the live backend (:7770) via the host LAN IP" {
# host.containers.internal is a slirp4netns-only name. The bridge
# gateway (10.89.3.1) lives in the rootless netns, NOT the host netns
# where llama-server runs -- it refuses. The host's LAN IP reaches it
# through the rootless NAT (verified from the container, 2026-08-22).
grep -qE '^Environment=OPENAI_API_BASE_URL=http://192\.168\.0\.143:52625' \
# 2026-08-22: frank-default-large moved from :52625 to :7770.
grep -qE '^Environment=OPENAI_API_BASE_URL=http://192\.168\.0\.143:7770' \
"$QDIR/open-webui.container" \
|| { echo "open-webui.container must set OPENAI_API_BASE_URL to the vLLM at 192.168.0.143:52625"; return 1; }
|| { echo "open-webui.container must set OPENAI_API_BASE_URL to the LLM at 192.168.0.143:7770"; return 1; }
}
@test "open-webui persists its data under /home/dank/Containers/open-webui/data" {