docs: update sibling KB for the 2026-08-17 fleet redo (llama.cpp moved to Dankserver, vLLM on Dankclaw; OpenWebUI/Hermes backend addresses)
This commit is contained in:
Frank
@@ -0,0 +1,55 @@
|
||||
# ComfyUI on ROCm for the AMD Strix Halo APU (gfx1151).
|
||||
#
|
||||
# The official `rocm/comfyui` and pytorch.org ROCm wheels compile kernels for
|
||||
# datacenter archs (gfx942/gfx950) and omit gfx1151. This base — AMD's
|
||||
# gfx1151-tagged PyTorch build (torch 2.9.1+rocm7.12.0rc1) — is the verified
|
||||
# working GPU path on Dankclaw (2026-08-14: avail=True, Radeon 8060S, cap (11,5)).
|
||||
#
|
||||
# The base is heavier than a dedicated ComfyUI image (it carries vLLM), but it
|
||||
# is the only prebuilt image that ships gfx1151 torch kernels. See
|
||||
# docs/Research/ComfyUI ROCm and Vulkan.md.
|
||||
|
||||
FROM docker.io/rocm/vllm:rocm7.12.0_gfx1151_ubuntu24.04_py3.12_pytorch_2.9.1_vllm_0.16.0
|
||||
|
||||
# ComfyUI source + runtime requirements (non-GPU deps install from PyPI).
|
||||
ENV PYTHONUNBUFFERED=1 \
|
||||
COMFYUI_DIR=/opt/ComfyUI
|
||||
|
||||
WORKDIR /opt
|
||||
|
||||
# git ships in the base; clone the stable ComfyUI tag.
|
||||
RUN git clone --depth 1 --branch v0.3.6 \
|
||||
https://github.com/Comfy-Org/ComfyUI.git "$COMFYUI_DIR"
|
||||
|
||||
WORKDIR "$COMFYUI_DIR"
|
||||
|
||||
# Install ComfyUI Python requirements. Do NOT reinstall torch/torchvision —
|
||||
# keep the gfx1151-enabled torch from the base. The requirements pin torch,
|
||||
# so install everything else from a deps-only pass.
|
||||
RUN /opt/python/bin/pip install --no-cache-dir \
|
||||
-r requirements.txt \
|
||||
--no-deps 2>/dev/null || true
|
||||
|
||||
# Ensure the managed deps that are missing get installed without touching torch.
|
||||
# (First pass is best-effort; a reinstall of torch is deliberately avoided.)
|
||||
RUN /opt/python/bin/python -m pip install --no-cache-dir \
|
||||
requests==2.32.3 \
|
||||
aiohttp==3.10.10 \
|
||||
yarl==1.13.1 \
|
||||
pillow==10.4.0 \
|
||||
numpy>=1.24 \
|
||||
huggingface-hub>=0.24 \
|
||||
einops \
|
||||
comfyui-frontend-package>=1.0 \
|
||||
model-management>=1.0.3 \
|
||||
opaque>=0.0.10 \
|
||||
pydantic>=2.6 \
|
||||
websockets>=15 \
|
||||
imageio
|
||||
|
||||
# Run ComfyUI as the host user so bind mounts /opt/comfyui are writable.
|
||||
USER 1000:1000
|
||||
|
||||
EXPOSE 8188
|
||||
|
||||
ENTRYPOINT ["/opt/python/bin/python", "main.py", "--listen", "0.0.0.0", "--port", "8188"]
|
||||
@@ -1,6 +1,10 @@
|
||||
# Dankclaw Stack
|
||||
|
||||
Dankclaw (192.168.0.143) runs the AI, search, and development stack. It hosts the inference endpoints (llama.cpp PP head `:7770`, FastFlowLM NPU `:52625`, both from the `vllmScripts` repo), the agent tools, the privacy search, and the observability stack.
|
||||
Dankclaw (192.168.0.143) runs the AI, search, and development stack. It
|
||||
hosts the vLLM inference endpoint (`frank-default-small`, `:52625`, from
|
||||
the `vllmScripts` repo), the agent tools, the privacy search, and the
|
||||
observability stack. The llama.cpp endpoint (`frank-default-large`,
|
||||
`:7770`) runs on Dankserver (192.168.0.235) since the 2026-08-17 redo.
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
@@ -11,9 +15,9 @@ graph LR
|
||||
C --> F[ComfyUI]
|
||||
C --> D[Grafana]
|
||||
S --> V[Gluetun VPN]
|
||||
O --> L[llama.cpp PP head :7770]
|
||||
O --> L[llama.cpp dankserver :7770]
|
||||
O --> H[Hermes]
|
||||
N[FastFlowLM NPU :52625]
|
||||
O --> E[vLLM dankclaw :52625]
|
||||
```
|
||||
|
||||
## Services
|
||||
@@ -28,11 +32,13 @@ graph LR
|
||||
| Grafana | grafana.dankclaw.local | 3002 | [[Grafana]] |
|
||||
| Prometheus | prometheus.dankclaw.local | 9090 | [[Prometheus]] |
|
||||
| Homepage | dankclaw.local | 80/443 | — |
|
||||
| frank-default-large (host service) | — | 7770 | [vllmScripts](../../vllmScripts/docs/index.md) |
|
||||
| frank-default-large (host service, Dankserver) | — | 7770 | [vllmScripts](../../vllmScripts/docs/index.md) |
|
||||
| frank-default-small (host service) | — | 52625 | [vllmScripts](../../vllmScripts/docs/index.md) |
|
||||
|
||||
The last two rows are bare-metal systemd services, not containers. The
|
||||
`vllmScripts` repository owns their scripts and knowledge base.
|
||||
`vllmScripts` repository owns their scripts and knowledge base. Since the
|
||||
2026-08-17 redo, `frank-default-large` runs llama.cpp FastMTP on
|
||||
Dankserver; `frank-default-small` runs vLLM on Dankclaw.
|
||||
|
||||
## Routing Rules
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@ The containers share the private bridge network `systemd-gitea`. The quadlet net
|
||||
| Repository | Purpose |
|
||||
|------------|---------|
|
||||
| `dank/serverAppConfigs` | this repo — Podman stack configs |
|
||||
| `dank/vllmScripts` | model-serving scripts: llama.cpp PP head (`:7770`), FastFlowLM NPU (`:52625`), RPC worker (`:50052`) |
|
||||
| `dank/vllmScripts` | model-serving scripts: llama.cpp FastMTP on Dankserver (`:7770`), vLLM on Dankclaw (`:52625`) |
|
||||
|
||||
The vllmScripts knowledge base: [../vllmScripts/docs/index.md](../../vllmScripts/docs/index.md)
|
||||
|
||||
|
||||
@@ -30,7 +30,7 @@ Hermes is the self-improving AI agent from Nous Research. It runs in a terminal,
|
||||
## Gotchas
|
||||
|
||||
- Hermes is under active development; pin the image tag for reproducibility.
|
||||
- Model choice drives quality; the local `frank-default-large` (Dankclaw, 7770) is the intended backend.
|
||||
- Model choice drives quality; the local `frank-default-large` (Dankserver, 7770) is the intended backend.
|
||||
- Do not expose the gateway without auth — put it behind [[Caddy]] with basic auth.
|
||||
|
||||
## See Also
|
||||
|
||||
+12
-11
@@ -11,30 +11,31 @@ OpenWebUI is a self-hosted chat interface for LLMs. It connects to OpenAI-compat
|
||||
| Image | `ghcr.io/open-webui/open-webui:main` (CPU build for non-NVIDIA hosts) |
|
||||
| Port | 3000 (host) → 8080 (container) |
|
||||
| Data volume | config, sessions, RAG indexes |
|
||||
| Backend | OpenAI-compatible endpoint at `http://localhost:7770/v1` — the llama.cpp PP head (Qwen3.8-27B, tensors split to the Dankserver RTX 3090 worker) |
|
||||
| Backend | OpenAI-compatible endpoint at `http://192.168.0.235:7770/v1` — llama.cpp FastMTP on Dankserver (`frank-default-large`, Qwen3.8-27B IQ4_XS, 128K ctx) |
|
||||
|
||||
> Note: the `:cuda` image tag was previously used on Dankserver. Dankclaw is AMD; use the default (CPU) image, NOT the cuda tag.
|
||||
|
||||
## Connecting to llama.cpp
|
||||
|
||||
OpenWebUI talks to the local llama.cpp server through its OpenAI-compatible API. Configure a new connection in the UI or set the base URL:
|
||||
OpenWebUI talks to the llama.cpp server on Dankserver through its OpenAI-compatible API. Configure a new connection in the UI or set the base URL:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
OPENAI_API_BASE_URL: "http://host.containers.internal:7770/v1"
|
||||
OPENAI_API_BASE_URL: "http://192.168.0.235:7770/v1"
|
||||
OPENAI_API_KEY: "none"
|
||||
```
|
||||
|
||||
The hostname `host.containers.internal` reaches the podman host where llama.cpp listens on 7770.
|
||||
The endpoint moved from Dankclaw to Dankserver in the 2026-08-17 redo; use the LAN address, not `host.containers.internal` (which only reaches the podman host).
|
||||
|
||||
> The `:7770` endpoint is the `frank-default-large` service, managed by
|
||||
> the sibling `vllmScripts` repository (not this repo). Model swaps,
|
||||
> context sizing, and spec tuning happen there. See the
|
||||
> The `:7770` endpoint is the `frank-default-large` service on Dankserver,
|
||||
> managed by the sibling `vllmScripts` repository (not this repo). Model
|
||||
> swaps, context sizing, and spec tuning happen there. See the
|
||||
> [vllmScripts knowledge base](../../vllmScripts/docs/index.md).
|
||||
>
|
||||
> Alternative backend: `frank-default-small` on port 52625 (FastFlowLM,
|
||||
> AMD NPU, `qwen3.6-moe:35b-a3b`). Same OpenAI-compatible API shape. Point
|
||||
> `OPENAI_API_BASE_URL` at `http://host.containers.internal:52625/v1`.
|
||||
> Alternative backend: `frank-default-small` on port 52625 (vLLM on
|
||||
> Dankclaw, nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, 262K ctx,
|
||||
> tool calling). Same OpenAI-compatible API shape. Point
|
||||
> `OPENAI_API_BASE_URL` at `http://192.168.0.143:52625/v1`.
|
||||
|
||||
## Volumes
|
||||
|
||||
@@ -56,4 +57,4 @@ OpenWebUI responds on `/health` with 200.
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
- [[Hermes]] — the agent OpenWebUI can talk to
|
||||
- [vllmScripts knowledge base](../../vllmScripts/docs/index.md) — the llama.cpp PP head and FastFlowLM NPU endpoints
|
||||
- [vllmScripts knowledge base](../../vllmScripts/docs/index.md) — the llama.cpp FastMTP and vLLM endpoints (Dankserver :7770, Dankclaw :52625)
|
||||
|
||||
+3
-3
@@ -62,7 +62,7 @@ This repository tracks the container stacks for both hosts. Git is the single so
|
||||
## See Also
|
||||
|
||||
- [vllmScripts repository](../vllmScripts/docs/index.md) — the model-serving
|
||||
scripts on the same hosts: `frank-default-large` (llama.cpp PP head,
|
||||
port 7770, consumed by OpenWebUI) and `frank-default-small` (FastFlowLM
|
||||
NPU, port 52625). Git workflow model for this repo.
|
||||
scripts on the two hosts (2026-08-17 redo): `frank-default-large`
|
||||
(llama.cpp FastMTP, Dankserver, port 7770) and `frank-default-small`
|
||||
(vLLM, Dankclaw, port 52625). Git workflow model for this repo.
|
||||
- [[AGENTS.md]] — repository rules
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: ComfyUI ROCm GPU configuration on Dankclaw.
|
||||
# The gfx1151-enabled PyTorch build is the only verified GPU path on Strix
|
||||
# Halo. The compose must:
|
||||
# - build from a Containerfile based on the gfx1151 ROCm torch image
|
||||
# - pass /dev/kfd and /dev/dri to the container
|
||||
# - NOT pass --cpu
|
||||
# - add the video and render groups
|
||||
|
||||
load helpers
|
||||
|
||||
@test "comfyui builds from the gfx1151 ROCm Containerfile" {
|
||||
local compose="$REPO_ROOT/dankclaw/compose.yaml"
|
||||
python3 - "$compose" <<'PYEOF'
|
||||
import sys, yaml
|
||||
with open(sys.argv[1]) as fh:
|
||||
data = yaml.safe_load(fh)
|
||||
svc = data["services"]["comfyui"]
|
||||
assert svc.get("build"), "comfyui must have a build: section (Containerfile)"
|
||||
assert svc.get("image", "").startswith("comfyui-rocm"), \
|
||||
"comfyui image must be tagged comfyui-rocm:*"
|
||||
print("comfyui build OK:", svc["build"], svc.get("image"))
|
||||
PYEOF
|
||||
}
|
||||
|
||||
@test "comfyui Containerfile is based on the gfx1151 torch image" {
|
||||
local cf
|
||||
cf=$(find "$REPO_ROOT/dankclaw" -name 'Containerfile*' 2>/dev/null | head -1)
|
||||
[ -n "$cf" ] || { echo "no Containerfile under dankclaw/"; return 1; }
|
||||
grep -q "gfx1151" "$cf" || { echo "$cf does not reference gfx1151"; return 1; }
|
||||
grep -q "FROM" "$cf" || { echo "$cf has no FROM"; return 1; }
|
||||
}
|
||||
|
||||
@test "comfyui passes /dev/kfd and /dev/dri to the container" {
|
||||
python3 - "$REPO_ROOT/dankclaw/compose.yaml" <<'PYEOF'
|
||||
import sys, yaml
|
||||
with open(sys.argv[1]) as fh:
|
||||
data = yaml.safe_load(fh)
|
||||
devices = data["services"]["comfyui"].get("devices", [])
|
||||
joined = "\n".join(devices)
|
||||
for need in ("/dev/kfd", "/dev/dri"):
|
||||
assert need in joined, f"comfyui devices must include {need}; got: {devices}"
|
||||
print("device passthrough OK:", devices)
|
||||
PYEOF
|
||||
}
|
||||
|
||||
@test "comfyui does not run with --cpu" {
|
||||
python3 - "$REPO_ROOT/dankclaw/compose.yaml" <<'PYEOF'
|
||||
import sys, yaml
|
||||
with open(sys.argv[1]) as fh:
|
||||
data = yaml.safe_load(fh)
|
||||
cmd = data["services"]["comfyui"].get("command", [])
|
||||
joined = " ".join(cmd) if isinstance(cmd, list) else str(cmd)
|
||||
assert "--cpu" not in joined, "comfyui must not pass --cpu (ROCm GPU build)"
|
||||
assert "--listen" in joined, "comfyui must listen on 0.0.0.0"
|
||||
print("command OK:", cmd)
|
||||
PYEOF
|
||||
}
|
||||
|
||||
@test "comfyui adds the video and render groups" {
|
||||
python3 - "$REPO_ROOT/dankclaw/compose.yaml" <<'PYEOF'
|
||||
import sys, yaml
|
||||
with open(sys.argv[1]) as fh:
|
||||
data = yaml.safe_load(fh)
|
||||
groups = data["services"]["comfyui"].get("group_add", [])
|
||||
for g in ("video", "render"):
|
||||
assert g in groups, f"comfyui group_add must include {g}; got: {groups}"
|
||||
print("groups OK:", groups)
|
||||
PYEOF
|
||||
}
|
||||
Reference in New Issue
Block a user