docs: update sibling KB for the 2026-08-17 fleet redo (llama.cpp moved to Dankserver, vLLM on Dankclaw; OpenWebUI/Hermes backend addresses)
This commit is contained in:
1 parent
c81933ccb4
commit
1de43f6777
7 files changed
+153
-21
No files matched your search
@@ -1,6 +1,10 @@
|
||||
# Dankclaw Stack
|
||||
|
||||
Dankclaw (192.168.0.143) runs the AI, search, and development stack. It hosts the inference endpoints (llama.cpp PP head `:7770`, FastFlowLM NPU `:52625`, both from the `vllmScripts` repo), the agent tools, the privacy search, and the observability stack.
|
||||
Dankclaw (192.168.0.143) runs the AI, search, and development stack. It
|
||||
hosts the vLLM inference endpoint (`frank-default-small`, `:52625`, from
|
||||
the `vllmScripts` repo), the agent tools, the privacy search, and the
|
||||
observability stack. The llama.cpp endpoint (`frank-default-large`,
|
||||
`:7770`) runs on Dankserver (192.168.0.235) since the 2026-08-17 redo.
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
@@ -11,9 +15,9 @@ graph LR
|
||||
C --> F[ComfyUI]
|
||||
C --> D[Grafana]
|
||||
S --> V[Gluetun VPN]
|
||||
O --> L[llama.cpp PP head :7770]
|
||||
O --> L[llama.cpp dankserver :7770]
|
||||
O --> H[Hermes]
|
||||
N[FastFlowLM NPU :52625]
|
||||
O --> E[vLLM dankclaw :52625]
|
||||
```
|
||||
|
||||
## Services
|
||||
@@ -28,11 +32,13 @@ graph LR
|
||||
| Grafana | grafana.dankclaw.local | 3002 | [[Grafana]] |
|
||||
| Prometheus | prometheus.dankclaw.local | 9090 | [[Prometheus]] |
|
||||
| Homepage | dankclaw.local | 80/443 | — |
|
||||
| frank-default-large (host service) | — | 7770 | [vllmScripts](../../vllmScripts/docs/index.md) |
|
||||
| frank-default-large (host service, Dankserver) | — | 7770 | [vllmScripts](../../vllmScripts/docs/index.md) |
|
||||
| frank-default-small (host service) | — | 52625 | [vllmScripts](../../vllmScripts/docs/index.md) |
|
||||
|
||||
The last two rows are bare-metal systemd services, not containers. The
|
||||
`vllmScripts` repository owns their scripts and knowledge base.
|
||||
`vllmScripts` repository owns their scripts and knowledge base. Since the
|
||||
2026-08-17 redo, `frank-default-large` runs llama.cpp FastMTP on
|
||||
Dankserver; `frank-default-small` runs vLLM on Dankclaw.
|
||||
|
||||
## Routing Rules
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@ The containers share the private bridge network `systemd-gitea`. The quadlet net
|
||||
| Repository | Purpose |
|
||||
|------------|---------|
|
||||
| `dank/serverAppConfigs` | this repo — Podman stack configs |
|
||||
| `dank/vllmScripts` | model-serving scripts: llama.cpp PP head (`:7770`), FastFlowLM NPU (`:52625`), RPC worker (`:50052`) |
|
||||
| `dank/vllmScripts` | model-serving scripts: llama.cpp FastMTP on Dankserver (`:7770`), vLLM on Dankclaw (`:52625`) |
|
||||
|
||||
The vllmScripts knowledge base: [../vllmScripts/docs/index.md](../../vllmScripts/docs/index.md)
|
||||
|
||||
|
||||
@@ -30,7 +30,7 @@ Hermes is the self-improving AI agent from Nous Research. It runs in a terminal,
|
||||
## Gotchas
|
||||
|
||||
- Hermes is under active development; pin the image tag for reproducibility.
|
||||
- Model choice drives quality; the local `frank-default-large` (Dankclaw, 7770) is the intended backend.
|
||||
- Model choice drives quality; the local `frank-default-large` (Dankserver, 7770) is the intended backend.
|
||||
- Do not expose the gateway without auth — put it behind [[Caddy]] with basic auth.
|
||||
|
||||
## See Also
|
||||
|
||||
+12
-11
@@ -11,30 +11,31 @@ OpenWebUI is a self-hosted chat interface for LLMs. It connects to OpenAI-compat
|
||||
| Image | `ghcr.io/open-webui/open-webui:main` (CPU build for non-NVIDIA hosts) |
|
||||
| Port | 3000 (host) → 8080 (container) |
|
||||
| Data volume | config, sessions, RAG indexes |
|
||||
| Backend | OpenAI-compatible endpoint at `http://localhost:7770/v1` — the llama.cpp PP head (Qwen3.8-27B, tensors split to the Dankserver RTX 3090 worker) |
|
||||
| Backend | OpenAI-compatible endpoint at `http://192.168.0.235:7770/v1` — llama.cpp FastMTP on Dankserver (`frank-default-large`, Qwen3.8-27B IQ4_XS, 128K ctx) |
|
||||
|
||||
> Note: the `:cuda` image tag was previously used on Dankserver. Dankclaw is AMD; use the default (CPU) image, NOT the cuda tag.
|
||||
|
||||
## Connecting to llama.cpp
|
||||
|
||||
OpenWebUI talks to the local llama.cpp server through its OpenAI-compatible API. Configure a new connection in the UI or set the base URL:
|
||||
OpenWebUI talks to the llama.cpp server on Dankserver through its OpenAI-compatible API. Configure a new connection in the UI or set the base URL:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
OPENAI_API_BASE_URL: "http://host.containers.internal:7770/v1"
|
||||
OPENAI_API_BASE_URL: "http://192.168.0.235:7770/v1"
|
||||
OPENAI_API_KEY: "none"
|
||||
```
|
||||
|
||||
The hostname `host.containers.internal` reaches the podman host where llama.cpp listens on 7770.
|
||||
The endpoint moved from Dankclaw to Dankserver in the 2026-08-17 redo; use the LAN address, not `host.containers.internal` (which only reaches the podman host).
|
||||
|
||||
> The `:7770` endpoint is the `frank-default-large` service, managed by
|
||||
> the sibling `vllmScripts` repository (not this repo). Model swaps,
|
||||
> context sizing, and spec tuning happen there. See the
|
||||
> The `:7770` endpoint is the `frank-default-large` service on Dankserver,
|
||||
> managed by the sibling `vllmScripts` repository (not this repo). Model
|
||||
> swaps, context sizing, and spec tuning happen there. See the
|
||||
> [vllmScripts knowledge base](../../vllmScripts/docs/index.md).
|
||||
>
|
||||
> Alternative backend: `frank-default-small` on port 52625 (FastFlowLM,
|
||||
> AMD NPU, `qwen3.6-moe:35b-a3b`). Same OpenAI-compatible API shape. Point
|
||||
> `OPENAI_API_BASE_URL` at `http://host.containers.internal:52625/v1`.
|
||||
> Alternative backend: `frank-default-small` on port 52625 (vLLM on
|
||||
> Dankclaw, nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, 262K ctx,
|
||||
> tool calling). Same OpenAI-compatible API shape. Point
|
||||
> `OPENAI_API_BASE_URL` at `http://192.168.0.143:52625/v1`.
|
||||
|
||||
## Volumes
|
||||
|
||||
@@ -56,4 +57,4 @@ OpenWebUI responds on `/health` with 200.
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
- [[Hermes]] — the agent OpenWebUI can talk to
|
||||
- [vllmScripts knowledge base](../../vllmScripts/docs/index.md) — the llama.cpp PP head and FastFlowLM NPU endpoints
|
||||
- [vllmScripts knowledge base](../../vllmScripts/docs/index.md) — the llama.cpp FastMTP and vLLM endpoints (Dankserver :7770, Dankclaw :52625)
|
||||
Reference in new issue
Block a user