dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Move frank-default-large from :52625 to :7770 (dankclaw)

Browse Source 
Operator request (2026-08-22): the primary inference server
(frank-default-large, the 27B CIRU model on Dankclaw) moves to the
standard llama.cpp port 7770. Both LLM services now listen on :7770,
distinguished by host (dankserver vs dankclaw).

Functional changes:
- scripts/frank-default-large.env: LISTEN_PORT 52625 -> 7770
- scripts/serve-frank-large.sh: default LISTEN_PORT 52625 -> 7770
- scripts/deploy.sh: LLM_HEALTH[dankclaw] -> :7770
- open-webui quadlet: OPENAI_API_BASE_URL -> 192.168.0.143:7770
  (re-pointed so chat keeps working; data dir preserved)
- scripts/bench-fast-greedy.sh: PORT 52625 -> 7770

Tests updated to the new topology (both LLM endpoints on :7770 by host;
open-webui -> :7770). Docs + pi KB updated; dated historical runbooks
keep their point-in-time ports. models.json (pi client) re-pointed to
:7770 separately, effective on the next client restart.
This commit is contained in:
Frank committed 2026-08-22 22:11:53 -07:00
1 parent e367f0f43c
commit fbfbc5c6db
15 files changed
+51 -45
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+5 -5
View File
@@ -27,7 +27,7 @@ host.
| Target | Runtime | Service | Model | Port | | Target | Runtime | Service | Model | Port |
|--------|---------|---------|-------|------| |--------|---------|---------|-------|------|
| Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 | | Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
| Dankclaw | CIRU ROCmFPX llama.cpp, PromptForge v3 runtime (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3 (Q8-FFNEQ-LateQ6, full-W8 views, Output-K8 MTP proxy, BF16 vision), 262K context / 1 slot | 52625 | | Dankclaw | CIRU ROCmFPX llama.cpp, PromptForge v3 runtime (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3 (Q8-FFNEQ-LateQ6, full-W8 views, Output-K8 MTP proxy, BF16 vision), 262K context / 1 slot | 7770 |
History: the 2026-08-17 redo retired the pipeline-parallel split and the History: the 2026-08-17 redo retired the pipeline-parallel split and the
FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
@@ -71,7 +71,7 @@ Use `scripts/deploy.sh` to deploy committed changes to a target. One script
covers both workloads per host: the podman stack and the LLM service. covers both workloads per host: the podman stack and the LLM service.
``` ```
./scripts/deploy.sh dankclaw # AI stack + frank-default-large (:52625) ./scripts/deploy.sh dankclaw # AI stack + frank-default-large (:7770)
./scripts/deploy.sh dankserver # media stack + frank-default-small (:7770) ./scripts/deploy.sh dankserver # media stack + frank-default-small (:7770)
./scripts/deploy.sh dankclaw --dry-run # preview without changes ./scripts/deploy.sh dankclaw --dry-run # preview without changes
``` ```
@@ -147,11 +147,11 @@ the bench finishes.
1. Stop the service before any benchmark: 1. Stop the service before any benchmark:
`sudo systemctl stop frank-default-small` (Dankserver) or `sudo systemctl stop frank-default-small` (Dankserver) or
`sudo systemctl stop frank-default-large` (Dankclaw). `sudo systemctl stop frank-default-large` (Dankclaw).
2. Run the bench on a scratch port (never the service ports 7770/52625). 2. Run the bench on a scratch port (never the LLM service port 7770).
3. Restart the service when the bench completes: `sudo systemctl start <svc>`. 3. Restart the service when the bench completes: `sudo systemctl start <svc>`.
4. Verify health before reporting: 4. Verify health before reporting:
`curl http://192.168.0.235:7770/health` or `curl http://192.168.0.235:7770/health` or
`curl http://192.168.0.143:52625/health`. `curl http://192.168.0.143:7770/health`.
## Pi Model Config (`~/.pi/agent/models.json`) ## Pi Model Config (`~/.pi/agent/models.json`)
@@ -159,7 +159,7 @@ The Pi agent reaches the fleet through `/home/dank/.pi/agent/models.json`.
This file lives outside the repository. Each provider is one host endpoint. This file lives outside the repository. Each provider is one host endpoint.
Each model `id` is the served alias. The two aliases are Each model `id` is the served alias. The two aliases are
`frank-default-small` (Dankserver `:7770`) and `frank-default-large` `frank-default-small` (Dankserver `:7770`) and `frank-default-large`
(Dankclaw `:52625`). (Dankclaw `:7770`).
The model `id` values are stable. The file The model `id` values are stable. The file
`/home/dank/.pi/agent/settings.json` references them. Two resolution rules `/home/dank/.pi/agent/settings.json` references them. Two resolution rules
README.md
+2 -2
View File
@@ -13,7 +13,7 @@ one deploy script.
| Host | User | IP | Podman stack | LLM service | LLM port | | Host | User | IP | Podman stack | LLM service | LLM port |
|------|------|----|--------------|-------------|----------| |------|------|----|--------------|-------------|----------|
| dankserver | dank | 192.168.0.235 | media and library | frank-default-small (llama.cpp, Qwen3.6-35B-A3B) | 7770 | | dankserver | dank | 192.168.0.235 | media and library | frank-default-small (llama.cpp, Qwen3.6-35B-A3B) | 7770 |
| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-default-large (CIRU ROCmFPX, Qwen3.8-27B) | 52625 | | dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-default-large (CIRU ROCmFPX, Qwen3.8-27B) | 7770 |
## Active Services ## Active Services
@@ -30,7 +30,7 @@ one deploy script.
| Gitea | dankclaw | gitea.dankclaw.local | 3001 | | Gitea | dankclaw | gitea.dankclaw.local | 3001 |
| Grafana | dankclaw | grafana.dankclaw.local | 3002 | | Grafana | dankclaw | grafana.dankclaw.local | 3002 |
| Prometheus | dankclaw | prometheus.dankclaw.local | 9090 | | Prometheus | dankclaw | prometheus.dankclaw.local | 9090 |
| frank-default-large | dankclaw | — (bare endpoint) | 52625 | | frank-default-large | dankclaw | — (bare endpoint) | 7770 |
## Structure ## Structure
dankclaw/open-webui/quadlet/open-webui.container
+3 -3
View File
@@ -5,8 +5,8 @@
# #
# Chat UI for the LLM fleet. It runs on the CPU image (dankclaw is CPU-only, so # Chat UI for the LLM fleet. It runs on the CPU image (dankclaw is CPU-only, so
# the :cuda image would waste a large pull for a GPU it cannot use) and points # the :cuda image would waste a large pull for a GPU it cannot use) and points
# its OpenAI connection at the live vLLM backend on this host via the bridge # its OpenAI connection at the live llama.cpp backend on this host via the
# gateway (192.168.0.143:52625, model frank-default-large). It publishes 3005 # host LAN IP (192.168.0.143:7770, model frank-default-large). It publishes 3005
# (3000 is Gitea's HTTP port). Data persists under # (3000 is Gitea's HTTP port). Data persists under
# /home/dank/Containers/open-webui/data. See docs/Dankclaw/OpenWebUI.md. # /home/dank/Containers/open-webui/data. See docs/Dankclaw/OpenWebUI.md.
# #
@@ -29,7 +29,7 @@ ContainerName=open-webui
Network=open-webui.network Network=open-webui.network
Image=ghcr.io/open-webui/open-webui:main Image=ghcr.io/open-webui/open-webui:main
PublishPort=3005:8080 PublishPort=3005:8080
Environment=OPENAI_API_BASE_URL=http://192.168.0.143:52625 Environment=OPENAI_API_BASE_URL=http://192.168.0.143:7770
Environment=ANONYMIZED_TELEMETRY=False Environment=ANONYMIZED_TELEMETRY=False
Environment=DO_NOT_TRACK=True Environment=DO_NOT_TRACK=True
Volume=/home/dank/Containers/open-webui/data:/app/backend/data Volume=/home/dank/Containers/open-webui/data:/app/backend/data
docs/Dankclaw/Dankclaw Stack.md
+3 -3
View File
@@ -1,7 +1,7 @@
# Dankclaw Stack # Dankclaw Stack
Dankclaw (192.168.0.143) runs the AI, search, and development stack. It Dankclaw (192.168.0.143) runs the AI, search, and development stack. It
hosts the LLM inference endpoint (`frank-default-large`, `:52625`, CIRU hosts the LLM inference endpoint (`frank-default-large`, `:7770`, CIRU
ROCmFPX llama.cpp), the agent tools, the privacy search, and the ROCmFPX llama.cpp), the agent tools, the privacy search, and the
observability stack. The llama.cpp endpoint (`frank-default-small`, observability stack. The llama.cpp endpoint (`frank-default-small`,
`:7770`) runs on Dankserver (192.168.0.235). `:7770`) runs on Dankserver (192.168.0.235).
@@ -17,7 +17,7 @@ graph LR
S --> V[Gluetun VPN] S --> V[Gluetun VPN]
O --> L[llama.cpp dankserver :7770] O --> L[llama.cpp dankserver :7770]
O --> H[Hermes] O --> H[Hermes]
O --> E[CIRU llama.cpp :52625] O --> E[CIRU llama.cpp :7770]
``` ```
## Services ## Services
@@ -32,7 +32,7 @@ graph LR
| Grafana | grafana.dankclaw.local | 3002 | [[Grafana]] | | Grafana | grafana.dankclaw.local | 3002 | [[Grafana]] |
| Prometheus | prometheus.dankclaw.local | 9090 | [[Prometheus]] | | Prometheus | prometheus.dankclaw.local | 9090 | [[Prometheus]] |
| Homepage | dankclaw.local | 80/443 | — | | Homepage | dankclaw.local | 80/443 | — |
| frank-default-large (host service, Dankclaw) | — | 52625 | [[deployment]] | | frank-default-large (host service, Dankclaw) | — | 7770 | [[deployment]] |
| frank-default-small (host service, Dankserver) | — | 7770 | [[deployment]] | | frank-default-small (host service, Dankserver) | — | 7770 | [[deployment]] |
The last two rows are bare-metal systemd services, not containers. This The last two rows are bare-metal systemd services, not containers. This
docs/Dankclaw/OpenWebUI.md
+5 -5
View File
@@ -11,19 +11,19 @@ OpenWebUI is a self-hosted chat interface for LLMs. It connects to OpenAI-compat
| Image | `ghcr.io/open-webui/open-webui:main` (CPU build; Dankclaw is CPU-only) | | Image | `ghcr.io/open-webui/open-webui:main` (CPU build; Dankclaw is CPU-only) |
| Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. | | Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
| Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` | | Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
| Default backend | vLLM on Dankclaw `:52625` = `frank-default-large` (via host LAN IP `192.168.0.143:52625`) | | Default backend | CIRU llama.cpp on Dankclaw `:7770` = `frank-default-large` (via host LAN IP `192.168.0.143:7770`) |
| Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) | | Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) |
| Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` | | Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` |
| Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` | | Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` |
## The LLM backends ## The LLM backends
The fleet runs two OpenAI-compatible endpoints. This mapping is the **live** state (verified 2026-08-19); it is the reverse of the 2026-08-17 note, which is now out of date. The fleet runs two OpenAI-compatible endpoints. This mapping is the **live** state (verified 2026-08-22). Both services listen on `:7770`, distinguished by host. On 2026-08-22 `frank-default-large` moved from `:52625` to `:7770`.
| Endpoint | Model name | Host | Capability | | Endpoint | Model name | Host | Capability |
|----------|-----------|------|-----------| |----------|-----------|------|-----------|
| `:52625` (vLLM) | `frank-default-large` | Dankclaw | completion | | `:7770` (CIRU llama.cpp) | `frank-default-large` | Dankclaw | completion |
| `:7770` (llama.cpp FastMTP) | `frank-default-small` | Dankserver | completion, multimodal | | `:7770` (llama.cpp) | `frank-default-small` | Dankserver | completion, multimodal |
The default connection points at the local vLLM (`frank-default-large`), so chat stays on-host. To add the other backend, open **Admin → Connections** in the UI and point an OpenAI connection at `http://192.168.0.235:7770/v1` (key `none`). The default connection points at the local vLLM (`frank-default-large`), so chat stays on-host. To add the other backend, open **Admin → Connections** in the UI and point an OpenAI connection at `http://192.168.0.235:7770/v1` (key `none`).
@@ -61,7 +61,7 @@ The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start
- Use the non-CUDA image on this CPU-only host. - Use the non-CUDA image on this CPU-only host.
- The default admin account is created on first login; secure it immediately. - The default admin account is created on first login; secure it immediately.
- If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready. - If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
- The LLM is reached via the host LAN IP `192.168.0.143:52625`, not the bridge gateway: the gateway (`10.89.3.1`) lives in the rootless netns, while llama-server runs in the host netns, so the gateway refuses. `host.containers.internal` is slirp4netns-only and does not resolve on a bridge either. To reach the Dankserver backend, use the LAN address `192.168.0.235:7770`. - The LLM is reached via the host LAN IP `192.168.0.143:7770`, not the bridge gateway: the gateway (`10.89.3.1`) lives in the rootless netns, while llama-server runs in the host netns, so the gateway refuses. `host.containers.internal` is slirp4netns-only and does not resolve on a bridge either. To reach the Dankserver backend, use the LAN address `192.168.0.235:7770`.
## History ## History
docs/LLM/Dankclaw System Specs.md
+1 -1
View File
@@ -93,7 +93,7 @@ cable between claw and server is required for 10G.
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM. - Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
- The GPU budget is the rocm-smi figure: 64 GiB VRAM reported by rocm-smi (SVM-backed). The older 96 GiB figure was the addressable SVM pool, not the budget. Trust rocm-smi and the runtime; do not use `free` or `dmidecode` for the GPU budget. - The GPU budget is the rocm-smi figure: 64 GiB VRAM reported by rocm-smi (SVM-backed). The older 96 GiB figure was the addressable SVM pool, not the budget. Trust rocm-smi and the runtime; do not use `free` or `dmidecode` for the GPU budget.
- The active service is `frank-default-large` on port 52625: CIRU ROCmFPX llama.cpp serving jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (262144 ctx, native MTP, PromptForge DualView). See [[ciru-activefpx-promptforge]]. - The active service is `frank-default-large` on port 7770 (moved from 52625 on 2026-08-22): CIRU ROCmFPX llama.cpp serving jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (262144 ctx, native MTP, PromptForge DualView). See [[ciru-activefpx-promptforge]].
- The runtime requires ROCm 7.15 (model card requirement). The system ROCm is 7.2.4; the retired ling3-ciru venv bundles a self-contained TheRock ROCm 7.15 SDK that the serve script puts first in `LD_LIBRARY_PATH`. - The runtime requires ROCm 7.15 (model card requirement). The system ROCm is 7.2.4; the retired ling3-ciru venv bundles a self-contained TheRock ROCm 7.15 SDK that the serve script puts first in `LD_LIBRARY_PATH`.
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18. - The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
- 1.1 TB free disk space is ample for model storage. - 1.1 TB free disk space is ample for model storage.
docs/LLM/ciru-activefpx-promptforge.md
+3 -3
View File
@@ -19,7 +19,7 @@ timestamp: 2026-08-22
> decision). Commit `c60156f` restored the v3 script pair. See > decision). Commit `c60156f` restored the v3 script pair. See
> [[kairic-edge]] for the retired experiment reference. > [[kairic-edge]] for the retired experiment reference.
`frank-default-large` (port 52625) serves `frank-default-large` (port 7770, moved from 52625 on 2026-08-22) serves
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` **v3** through the CIRU `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` **v3** through the CIRU
ROCmFPX llama.cpp runtime. This replaced the retired vLLM service that ROCmFPX llama.cpp runtime. This replaced the retired vLLM service that
previously occupied the slot. The v2 deployment moved to v3 on 2026-08-21. previously occupied the slot. The v2 deployment moved to v3 on 2026-08-21.
@@ -150,7 +150,7 @@ against the model card before the service restart.
## Service ## Service
- Unit: `frank-default-large.service`, port 52625 - Unit: `frank-default-large.service`, port 7770
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-default-large.env` - Script: `scripts/serve-frank-large.sh`, env `scripts/frank-default-large.env`
- The script exports the ROCm 7.15 SDK libraries in `LD_LIBRARY_PATH` before - The script exports the ROCm 7.15 SDK libraries in `LD_LIBRARY_PATH` before
the system ROCm. `ROCM_PATH` points at the devel SDK. the system ROCm. `ROCM_PATH` points at the devel SDK.
@@ -197,7 +197,7 @@ it and generation proceeds.
## Operational Notes ## Operational Notes
- Health: `curl http://127.0.0.1:52625/health`. Returns 503 while loading. - Health: `curl http://127.0.0.1:7770/health`. Returns 503 while loading.
- Model load is slow (v3 five-file payload ~36.7 GiB); do not kill the - Model load is slow (v3 five-file payload ~36.7 GiB); do not kill the
process during warmup. The page cache makes a repeat load fast right after process during warmup. The page cache makes a repeat load fast right after
download, but a later cold start takes several minutes. download, but a later cold start takes several minutes.
docs/LLM/deployment.md
+4 -4
View File
@@ -18,7 +18,7 @@ repository to a running service.
| Target | Host | Runtime | Service | Port | | Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------| |--------|------|---------|---------|------|
| Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 | | Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 52625 | | Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 7770 |
## Workflow ## Workflow
@@ -87,7 +87,7 @@ not used.
Health endpoints: Health endpoints:
- Dankserver: `http://127.0.0.1:7770/health` - Dankserver: `http://127.0.0.1:7770/health`
- Dankclaw: `http://127.0.0.1:52625/health` - Dankclaw: `http://127.0.0.1:7770/health`
## Manual Update ## Manual Update
@@ -114,7 +114,7 @@ git pull origin main
sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service sudo ln -sf /home/dank/serverAppConfigs/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sudo systemctl daemon-reload sudo systemctl daemon-reload
sudo systemctl restart frank-default-large sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:52625/health curl -sf http://127.0.0.1:7770/health
``` ```
## Service Units ## Service Units
@@ -137,7 +137,7 @@ launches through `scripts/serve-frank-small.sh`, which reads tunables from
The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v3 five-file payload: GGUF `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v3 five-file payload: GGUF
+ FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj) under the alias + FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj) under the alias
`frank-default-large` on port 52625. It runs `frank-default-large` on port 7770 (moved from 52625 on 2026-08-22). It runs
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP 7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP
(`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context (`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context
docs/LLM/index.md
+2 -2
View File
@@ -26,14 +26,14 @@ ngram-mod, 36 MoE experts on CPU, 700K ctx]
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
D[gfx1151 iGPU 64 GiB unified memory] D[gfx1151 iGPU 64 GiB unified memory]
E[CIRU ROCmFPX llama.cpp PromptForge v3 E[CIRU ROCmFPX llama.cpp PromptForge v3
frank-default-large, port 52625] frank-default-large, port 7770]
F[jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3 F[jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3
262K ctx / 1 slot, full-W8 views, Output-K8 MTP, BF16 vision] 262K ctx / 1 slot, full-W8 views, Output-K8 MTP, BF16 vision]
D --> E --> F D --> E --> F
end end
I[LLM Clients / API] -->|HTTP :7770| B I[LLM Clients / API] -->|HTTP :7770| B
I -->|HTTP :52625| E I -->|HTTP :7770| E
``` ```
## Servers ## Servers
scripts/bench-fast-greedy.sh
+1 -1
View File
@@ -7,7 +7,7 @@
set -euo pipefail set -euo pipefail
HOST="127.0.0.1" HOST="127.0.0.1"
PORT=52625 PORT=7770
ALIAS="frank-default-large" ALIAS="frank-default-large"
# A coding prompt representative of the card's 164-task suite. # A coding prompt representative of the card's 164-task suite.
scripts/deploy.sh
+2 -2
View File
@@ -21,7 +21,7 @@
# #
# Targets: # Targets:
# dankclaw - AI/search/dev/observability stack + frank-default-large # dankclaw - AI/search/dev/observability stack + frank-default-large
# (CIRU ROCmFPX llama.cpp, Qwen3.8-27B, port 52625) # (CIRU ROCmFPX llama.cpp, Qwen3.8-27B, port 7770)
# dankserver - media/library stack (bare containers) + frank-default-small # dankserver - media/library stack (bare containers) + frank-default-small
# (upstream llama.cpp, Qwen3.6-35B-A3B, port 7770) # (upstream llama.cpp, Qwen3.6-35B-A3B, port 7770)
# #
@@ -51,7 +51,7 @@ COPY_SHARED[dankclaw]=1
# picks up the new paths at its next manual start. # picks up the new paths at its next manual start.
LLM_SERVICES[dankclaw]="frank-default-large" LLM_SERVICES[dankclaw]="frank-default-large"
LLM_UNITS[dankclaw]="frank-default-large frank-default-npu" LLM_UNITS[dankclaw]="frank-default-large frank-default-npu"
LLM_HEALTH[dankclaw]="http://127.0.0.1:52625/health" LLM_HEALTH[dankclaw]="http://127.0.0.1:7770/health"
# CIRU llama.cpp (ROCm 7.15) loads the v3 five-file payload (GGUF + FFN + # CIRU llama.cpp (ROCm 7.15) loads the v3 five-file payload (GGUF + FFN +
# GDN + Output-K8 .pfs + mmproj, ~36.7 GiB) + MTP draft/Output-K8 proxy and # GDN + Output-K8 .pfs + mmproj, ~36.7 GiB) + MTP draft/Output-K8 proxy and
# initializes PromptForge views; allow 20 minutes. # initializes PromptForge views; allow 20 minutes.
scripts/frank-default-large.env
+4 -2
View File
@@ -6,7 +6,9 @@
# serving jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3). v3 tensor # serving jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3). v3 tensor
# allocation (FFN-equalized, late-Q6 writers, full-Q8 output) with the # allocation (FFN-equalized, late-Q6 writers, full-Q8 output) with the
# full-W8 PromptForge companions, the Output-K8 draft proxy, and the BF16 # full-W8 PromptForge companions, the Output-K8 draft proxy, and the BF16
# vision projector. Alias frank-default-large, port 52625. # vision projector. Alias frank-default-large, port 7770 (moved from 52625
# on 2026-08-22; the primary inference server sits on the standard
# llama.cpp port).
# #
# This runtime replaced the retired vLLM service that previously occupied # This runtime replaced the retired vLLM service that previously occupied
# this slot. See docs/ciru-activefpx-promptforge.md. # this slot. See docs/ciru-activefpx-promptforge.md.
@@ -41,7 +43,7 @@ ROCM_CORE=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/
ALIAS=frank-default-large ALIAS=frank-default-large
BIND_ADDR=0.0.0.0 BIND_ADDR=0.0.0.0
LISTEN_PORT=52625 LISTEN_PORT=7770
# Serving: full native 262144 context (n_ctx_train), 1 slot (user request # Serving: full native 262144 context (n_ctx_train), 1 slot (user request
# 2026-08-19: parallel slots degraded performance). -c is the TOTAL # 2026-08-19: parallel slots degraded performance). -c is the TOTAL
scripts/serve-frank-large.sh
+1 -1
View File
@@ -41,7 +41,7 @@ fi
: "${ROCM_CORE:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_core}" : "${ROCM_CORE:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_core}"
: "${ALIAS:=frank-default-large}" : "${ALIAS:=frank-default-large}"
: "${BIND_ADDR:=0.0.0.0}" : "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=52625}" : "${LISTEN_PORT:=7770}"
: "${CTX_SIZE:=524288}" : "${CTX_SIZE:=524288}"
: "${PARALLEL:=2}" : "${PARALLEL:=2}"
: "${BATCH_SIZE:=4096}" : "${BATCH_SIZE:=4096}"
tests/deployment_flow.bats
+9 -6
View File
@@ -55,17 +55,20 @@ load helpers
} }
# The unified deploy.sh must manage the LLM fleet services on both hosts: # The unified deploy.sh must manage the LLM fleet services on both hosts:
# frank-default-small (dankserver, :7770) and frank-default-large (dankclaw, # frank-default-small (dankserver) and frank-default-large (dankclaw). A
# :52625). A deploy that only restarts the podman stack leaves the LLM # deploy that only restarts the podman stack leaves the LLM units pointing
# units pointing at the retired vllmScripts checkout. # at the retired vllmScripts checkout.
@test "deploy.sh restarts the LLM services on both targets" { @test "deploy.sh restarts the LLM services on both targets" {
grep -q 'frank-default-small' "$REPO_ROOT/scripts/deploy.sh" grep -q 'frank-default-small' "$REPO_ROOT/scripts/deploy.sh"
grep -q 'frank-default-large' "$REPO_ROOT/scripts/deploy.sh" grep -q 'frank-default-large' "$REPO_ROOT/scripts/deploy.sh"
} }
@test "deploy.sh health-checks both LLM endpoints" { @test "deploy.sh health-checks both LLM endpoints (both on :7770, by host)" {
grep -qE '7770' "$REPO_ROOT/scripts/deploy.sh" # 2026-08-22: frank-default-large moved from :52625 to :7770 on
grep -qE '52625' "$REPO_ROOT/scripts/deploy.sh" # dankclaw. Both LLM services now listen on :7770, distinguished by
# host (dankserver vs dankclaw).
grep -qE '^LLM_HEALTH\[dankserver\]="http://127\.0\.0\.1:7770/health"' "$REPO_ROOT/scripts/deploy.sh"
grep -qE '^LLM_HEALTH\[dankclaw\]="http://127\.0\.0\.1:7770/health"' "$REPO_ROOT/scripts/deploy.sh"
} }
@test "deploy.sh pulls on the merged checkout on both targets" { @test "deploy.sh pulls on the merged checkout on both targets" {
tests/open_webui_quadlet.bats
+6 -5
View File
@@ -3,8 +3,8 @@
# #
# The old open-webui was a bare container on dankserver (port 3000) pointing at a # The old open-webui was a bare container on dankserver (port 3000) pointing at a
# dead LLM backend (:9090). This unit stands one up on dankclaw on a free port # dead LLM backend (:9090). This unit stands one up on dankclaw on a free port
# (3005), on the CPU image (dankclaw is CPU-only), wired to the live vLLM # (3005), on the CPU image (dankclaw is CPU-only), wired to the live
# backend (host.containers.internal:52625, model frank-default-large). # llama.cpp backend (192.168.0.143:7770, model frank-default-large).
load helpers load helpers
@@ -58,14 +58,15 @@ QDIR="$REPO_ROOT/dankclaw/open-webui/quadlet"
|| { echo "open-webui.container must publish 3005:8080"; return 1; } || { echo "open-webui.container must publish 3005:8080"; return 1; }
} }
@test "open-webui points its LLM at the live vLLM backend (52625) via the host LAN IP" { @test "open-webui points its LLM at the live backend (:7770) via the host LAN IP" {
# host.containers.internal is a slirp4netns-only name. The bridge # host.containers.internal is a slirp4netns-only name. The bridge
# gateway (10.89.3.1) lives in the rootless netns, NOT the host netns # gateway (10.89.3.1) lives in the rootless netns, NOT the host netns
# where llama-server runs -- it refuses. The host's LAN IP reaches it # where llama-server runs -- it refuses. The host's LAN IP reaches it
# through the rootless NAT (verified from the container, 2026-08-22). # through the rootless NAT (verified from the container, 2026-08-22).
grep -qE '^Environment=OPENAI_API_BASE_URL=http://192\.168\.0\.143:52625' \ # 2026-08-22: frank-default-large moved from :52625 to :7770.
grep -qE '^Environment=OPENAI_API_BASE_URL=http://192\.168\.0\.143:7770' \
"$QDIR/open-webui.container" \ "$QDIR/open-webui.container" \
|| { echo "open-webui.container must set OPENAI_API_BASE_URL to the vLLM at 192.168.0.143:52625"; return 1; } || { echo "open-webui.container must set OPENAI_API_BASE_URL to the LLM at 192.168.0.143:7770"; return 1; }
} }
@test "open-webui persists its data under /home/dank/Containers/open-webui/data" { @test "open-webui persists its data under /home/dank/Containers/open-webui/data" {